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PREFACE 


This  study  was  conducted  within  RAND's  Center  for  the  Study  of 
Food  and  Nutrition  Policy  as  part  of  RAND’s  Child  Nutrition  Analysis 
Project  (CNAP)  with  the  United  States  Department  of  Agriculture's 
Food  and  Nutrition  Service  under  “Obtaining  States’  Nutritional 
Monitoring  Data."  This  document  reports  on  a  study  of  state  nutri¬ 
tional  monitoring  data  and  should  be  of  benefit  to  those  interested  in 
the  operation  of  child  nutrition  programs. 

This  volume  contains  the  narrative  report.  An  appendix  volume 
(Liisa  Hiatt  and  Jacob  Alex  Klerman,  State  Monitoring  of  National 
School  Lunch  Program  (NSLP)  Nutritional  Content:  State-by- State 
Results,  RAND,  MR-1296/1-USDAFNS,  2001)  provides  the  detailed 
state  responses. 

The  main  text  of  this  report  was  written  in  the  fall  of  1999.  Since  that 
time,  there  have  been  some  changes — for  example,  the  Dietary 
Guidelines  for  Americans  were  updated  in  2000.  The  text  of  this  re¬ 
port  mentions  only  the  1995  update  because  the  2000  update  was  not 
completed  when  the  report  was  written. 
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SUMMARY 


INTRODUCTION 

As  part  of  the  School  Meals  Initiative  (SMI),  the  U.S.  Department  of 
Agriculture-Food  and  Nutrition  Service  (USDA-FNS)  now  requires 
each  state  to  regularly  review  the  nutritional  content  of  food  served 
by  each  School  Food  Authority  (SFA)  as  part  of  the  National  School 
Lunch  Program  (NSLP).  While  states  must  monitor  the  nutritional 
content  of  school  meals,  they  are  not  required  to  forward  any  infor¬ 
mation  to  USDA-FNS.  However,  USDA-FNS  is  required  to  measure 
progress  against  Objective  2.1  of  FSN’s  1997-2002  Strategic  Plan, 
which  states  that  it  will  “ensure  that  school  meals  are  consistent  with 
the  Dietary  Guidelines  for  Americans  [DGA]  and  the  Recommended 
Daily  Allowances  [RDA].”  In  September  2000,  FNS  issued  a  com¬ 
pletely  revised  Strategic  Plan,  which  established  goals  for  school 
lunches  under  a  new  Objective  1.3:  “Improved  nutritional  quality  of 
meals,  commodities,  and  other  program  benefits.”  The  “target”  es¬ 
tablished  under  Objective  1.3  is  “By  2005,  reach  less  than  or  equal  to 
30%  calories  from  total  fat  and  less  than  10%  calories  from  saturated 
fat;  maintain  calorie,  vitamin  and  mineral  content  at  greater  than  or 
equal  to  33%  of  RDA.”  To  measure  progress,  USDA-FNS  needs  to 
produce  state  and  national  aggregations  of  the  nutrient  content  in 
school  lunches  to  show  that  meals  in  a  given  state  or  in  the  country 
as  a  whole  are  consistent  with  the  DGA  and  RDA.  In  addition,  FNS 
needs  to  do  this  while  imposing  minimal  reporting  burdens  on  the 
states. 
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The  USDA-FNS  contracted  with  RAND  to  look  more  carefully  at  how 
some  states  are  performing  SMI  reviews  and  how  they  might  be  able 
to  transmit  data  with  limited  additional  burden.  In  particular,  it 
asked  seven  states  to  work  collaboratively  with  RAND — through  on¬ 
site,  in-depth  interviews — to  explore  possible  reporting  systems: 
California,  Georgia,  Nebraska,  New  Jersey,  New  York,  Texas,  and 
Wisconsin.  These  states  were  chosen  as  representative  of  regional 
differences  across  the  country  and  of  methods  for  conducting  SMI 
reviews.  This  report  describes  the  results  of  the  analysis  of  possible 
approaches  for  states  to  report  the  results  of  their  nutritional  reviews 
to  USDA-FNS. 

WHAT  DATA  SHOULD  BE  PROVIDED? 

To  design  an  appropriate  system,  USDA-FNS  must  determine  what 
information  from  the  SMI  reviews  it  needs,  while  again  minimizing 
the  burden  to  the  states.  Starting  with  an  exhaustive  list  of  data  ele¬ 
ments  to  collect,  we  worked  with  the  selected  states  to  identify  a  set 
of  required  elements;  we  also  identified  some  optional  elements  and 
some  elements  that  should  not  be  included.  The  three  sets  are 
summarized  in  Table  S.l. 


Table  S.l 

Summary  of  Data  Elements  to  Be  Required,  Optional,  and  Not  Included 


Required  Elements 

Optional  Elements 

Elements  Not 
Included 

Type  of  SEA 

SFA  identifier 

Name  of  reviewer 

Menu  planning  system  used 

Review  week 

Dates  of  review 

Analysis  lunch  only,  breakfast  only, 
or  lunch  and  breakfast 

Contact  person 

Standards 

Analysis  weighted? 

Classification  of  “any 
reasonable  approach” 

Review  done  by  grade  or  age 

Urban  or  rural  area 

Lowest  grade /age 

Size  of  district 

Highest  grade/ age 

Estimated  average  daily  participation 
Actual  output  from  analysis  software 

Comments 
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HOW  DO  STATES  CURRENTLY  PERFORM  SMI  REVIEWS? 

To  understand  each  state’s  ability  to  report  the  required  information 
and  to  develop  a  system  that  would  impose  the  minimum  burden  on 
the  states,  we  examined  how  the  states  currently  perform  SMI  re¬ 
views.  Overall,  the  seven  states  interviewed  followed  a  similar  pro¬ 
cess  in  which  the  SFA  gathers  information  for  review,  the  nutrient 
analysis  is  performed,  a  correction  plan  is  developed  with  the  SFA, 
and  the  review  is  completed.  However,  each  state  operates  slightly 
differently.  Table  S.2  summarizes  the  information  collected  on  how 
SMI  reviews  are  conducted  in  the  seven  states,  showing  the  variation. 
Whether  the  state  office  receives  the  data  and  whether  the  state  has 
compiled  it  into  any  type  of  central  database  are  particularly  perti¬ 
nent.  Except  for  Texas,  all  the  states  receive  the  data,  and  most  have 
databases. 

HOW  SHOULD  THE  DATA  BE  COLLECTED? 

Although  there  are  a  number  of  options  for  collecting  the  data,  given 
USDA-FNS’s  stated  goal  of  collecting  data  at  minimal  cost  to  the 
states,  it  is  appropriate  for  the  USDA-FNS  to  accept  the  data  in  any 
reasonable  format  the  states  are  willing  to  provide,  including  USDA 
file  formats,  other  file  formats  from  states  (assuming  the  formats 

Table  S.2 

Summary  of  How  SMI  Reviews  Are  Performed  in  the  Seven  States 


State 

Who  Does  Analysis 

Reviewer  Other 

Weighted 

Analysis? 

State  Has 

Data?* 

Software 

Used 

California 

X 

X 

Some 

Y,DB 

NutriKids 

Georgia 

X 

X 

Y 

Y,NDB 

NutriKids 

Nebraska 

X 

Y 

Y,DB 

NutriKids 

New  Jersey 

X 

Y 

Y,NDB 

NutriKids 

New  York 

X 

X 

Y 

Y,DB 

NutriKids 

Texas 

X 

N 

N 

NutriKids 

Wisconsin 

X 

X 

Y 

Y,  DB 

NutriKids 

*DB  =  database  or  spreadsheet;  NDB  =  no  database  or  spreadsheet 
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contain  all  the  required  data  elements),  and  through  an  Internet 
form.  USDA-FNS  should  also  consider  ensuring  that  whatever  for¬ 
mat  is  used,  the  states  will  be  able  to  customize  the  databases  for 
their  own  purposes.  Finally,  given  the  five-year  SMI  review  cycle, 
USDA-FNS  will  need  only  a  moderate-sized  database,  which  can  be 
run  on  the  standard  desktop  personal  computer  purchased  today 
(late  1999).  The  tabulations  required  are  straightforward  and  could 
be  precoded.  An  analyst  with  rudimentary  experience  in  statistical 
analysis  should  be  able  to  program  the  required  queries  without 
major  assistance. 

DISCUSSION  OF  THE  CONTENT  OF  THE  SMI  EFFORT 

We  also  considered  the  SMI  effort  more  broadly.  We  found  some 
positive  effects  of  the  current  review  procedures.  The  experience  of 
preparing  for  a  review  itself  raises  consciousness  of  the  nutritional  is¬ 
sues,  and  the  results  of  the  review  lead  to  improvements  in  the  nutri¬ 
tional  content  of  meals. 

We  also  found  three  issues  about  the  quality  of  the  data  collected. 
First,  states  have  concerns  about  inter-rater  reliability;  two  reviews  of 
the  same  meals  might  reach  different  conclusions.  Although  such 
problems  are  nearly  impossible  to  eliminate,  they  can  be  minimized 
by  promulgating  common  procedures  for  reviews,  providing  formal 
initial  and  ongoing  training  in  those  procedures,  and  initiating  a  pro¬ 
gram  to  test  the  quality  of  completed  reviews. 

Second,  even  if  the  reviews  are  perfect,  are  they  representative  of 
meals  at  the  SFA  level?  In  other  words,  how  much  information  do 
the  results  of  a  review  (of  meals  served  one  given  week  in  a  given 
school)  yield  about  all  the  meals  served  in  that  school  (or  SFA)?  To 
address  this  issue,  USDA-FNS  could  fund  a  small  pilot  study  that 
would  randomly  select  a  small  group  of  SFAs  and  collect  SMI  type 
information  across  schools  and  through  time. 

Third,  are  the  reviews  at  the  state  level  representative?  While  the 
USDA  Strategic  Plan  requires  information  about  all  meals  served,  re¬ 
view  processes  can  only  collect  information  about  meals  reviewed. 
Standard  arguments  from  statistical  sampling  imply  that  it  is  possi¬ 
ble  to  construct  a  statistically  accurate  portrait  of  all  meals  served  by 
a  group  of  SFAs  (e.g.,  a  state)  by  averaging  results  across  multiple  re- 


Summary  xv 


views  and  across  individual  weeks  from  individual  schools  from 
multiple  districts.  For  these  arguments  to  apply,  reviewed  meals 
must  be  selected  randomly,  but  the  current  procedures  do  not  guar¬ 
antee  random  selection. 

In  addition,  more  precise  estimates  could  be  obtained  with  a  differ¬ 
ent  review  cycle.  Currently,  one  review  is  required  per  SFA  in  each 
five-year  cycle,  regardless  of  district  size.  However,  SFAs  vary  in  size 
dramatically.  For  example,  about  half  the  meals  in  New  York  State 
are  served  by  the  New  York  City  public  school  system.  Statistical 
considerations  suggest  allocating  more  resources  to  large  SFAs  and 
fewer  resources  to  smaller  SFAs.^  Such  a  procedure  would  also  ease 
the  burden  on  smaller  SFAs,  probably  reducing  opposition  to  the 
SMI  effort. 


^Effective  allocation  of  the  sample  to  minimize  variance  across  states  as  a  whole  re¬ 
quires  more  resources  for  larger  SFAs. 
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Chapter  One 

INTRODUCTION 


As  part  of  the  School  Meals  Initiative,  the  U.S.  Department  of 
Agriculture-Food  and  Nutrition  Service  (USDA-FNS)  now  requires 
each  state  to  regularly  review  the  nutritional  content  of  food  served 
by  each  School  Food  Authority  (SFA)i  as  part  of  the  National  School 
Lunch  Program  (NSLP).  However,  the  data  collected  as  part  of  this 
system  of  reviewing  nutritional  content  are  not  currently  reported  to 
anyone  outside  of  the  state  in  which  they  are  collected.^  This  report 
describes  the  results  of  a  RAND  analysis  of  possible  approaches  for 
states  to  report  the  results  of  their  nutritional  reviews  to  USDA. 

POLICY  CONTEXT 

In  this  section,  we  briefly  review  the  history  of  the  National  School 
Lunch  Program,  the  School  Meals  Initiative  (SMI),  and  SMI  reviews. 

USDA-FNS  School  Nutrition  Programs 

The  NSLP  was  established  in  1946  by  the  National  School  Lunch  Act 
The  legislation’s  preamble  stated  its  goals: 


^An  SFA  is  usually  a  public  school  district,  a  private  school,  a  charter  school,  a  resi¬ 
dential  child  care  institution  (RCCI),  or  a  juvenile  detention  center. 

^At  the  time  this  report  was  written,  FNS  did  not  require  any  data  from  the  states. 
Subsequently,  some  of  the  regional  offices  of  FNS  have  required  states  to  submit  these 
data  quarterly.  Specifically,  New  York  reported  to  us  that  the  Northeast  Region  re¬ 
quires  quarterly  submissions. 
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as  a  measure  of  national  security,  to  safeguard  the  health  and  well¬ 
being  of  the  Nation’s  children  and  to  encourage  the  domestic  con¬ 
sumption  of  nutritious  agricultural  commodities  and  other  food,  by 
assisting  the  States,  through  grants-in-aid  and  other  means,  in  pro¬ 
viding  an  adequate  supply  of  foods  and  other  facilities  for  the  es¬ 
tablishment,  maintenance,  operation,  and  expansion  of  nonprofit 
school  lunch  programs.^ 

Through  subsequent  amendments  and  increased  funding,  the  pro¬ 
gram  has  expanded  to  95,000  public  and  private  schools  and  resi¬ 
dential  child  care  institutions.  More  than  26  million  children  receive 
free  or  reduced-price  lunches  every  day.  Roughly  92  percent  of 
schoolchildren  in  the  United  States  are  in  schools  with  lunch  pro¬ 
grams,  and  about  58  percent  participate  in  the  program  on  an  aver¬ 
age  school  day.  For  fiscal  year  1998,  the  program  had  a  budget  of 
$4.2  billion.4 

While  the  NSLP  was  intended  to  “safeguard  the  health  and  well-be¬ 
ing  of  the  Nation’s  children,”^  through  1996  the  program’s  nutri¬ 
tional  focus  was  on  broad  food  groups  and  total  calories.  In  those 
fifty  years,  the  nutritional  status  of  Americans  and  scientific  under¬ 
standing  of  proper  nutrition  both  changed.  Those  changes  were  re¬ 
flected  in  a  series  of  Dietary  Guidelines  for  Americans  (DGA)  begin¬ 
ning  in  1980  and  most  recently  in  1995.®  The  1990  DGA  and  the  Food 
Pyramid  that  was  developed  to  publicize  them  reflected  significant 
changes  in  the  content  of  the  guidelines.  The  language  of  the  new 
guidelines  was  more  positive,  was  oriented  toward  the  total  diet,  and 
provided  more  specific  information  regarding  food  selection.  For  the 
first  time,  numerical  recommendations  were  made  for  intakes  of 
dietary  fat  and  saturated  fat.^  These  changes  were  further  refined  in 
the  1995  DGA. 


^Section  2  of  the  National  School  Lunch  Act,  42  USC  1751. 

^From  http://www.fns.usda.gov/cnd/Lunch/AboutLunch/faqs.htm. 

^Section  2  of  the  National  School  Lunch  Act,  42  USC  1751. 

®When  this  report  was  written,  the  1995  DGA  were  the  most  recent.  The  2000  version 
has  since  come  out  but  is  not  referenced  in  this  report. 

^Dietary  Guidelines  Advisory  Committee,  1995,  Appendix  I:  History  of  Dietary  Guide¬ 
lines  for  Americans. 
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Concern  about  the  content  of  school  meals  and  their  role  in  promot¬ 
ing  the  nutrition  of  children  was  further  focused  by  the  1992  School 
Nutrition  Dietary  Assessment  (SNDA).  It  found  that  many  school 
meals  did  not  meet  the  then- current  1990  DGA.  The  study  showed 
that  the  average  school  lunch  had  27  percent  more  calories  from  fat, 
twice  as  much  sodium,  and  50  percent  more  calories  from  saturated 
fat  than  the  recommendations.®  In  addition,  SNDA  found  that 
children  who  ate  the  hot  school  lunch  were  consuming  significantly 
higher  numbers  of  calories  from  fat  than  children  who  obtained  their 
lunches  from  other  sources.® 

SCHOOL  MEALS  INITIATIVE  AND  THE  PROPOSED  RULE 

In  response,  public  hearings  were  held  and  USDA  created  a  plan  for 
the  NSLP  that  ultimately  became  the  School  Meals  Initiative  (SMI). 
From  the  public  comments,  USDA  developed  five  Guiding  Principles 
for  SMI: 

•  Healthy  Children:  Provide  school  meals  that  promote  health, 
prevent  disease,  and  meet  the  Dietary  Guidelines  for  Americans 
(DGA). 

•  Customer  Appeal:  Serve  meals  that  children  will  want  to  eat. 

•  Flexibility:  Reduce  paperwork,  streamline  reporting  systems, 
recognize  differences,  and  offer  different  methods  for  meeting 
NSLP  guidelines. 

•  Investing  in  People:  Provide  technical  assistance  to  schools  so 
that  they  can  improve  the  nutritional  content  of  school  meals. 

•  Building  Partnerships:  Create  partnerships  with  public  and  pri¬ 
vate  entities  to  ensure  continuing  collaborative  efforts. 

On  June  10, 1994,  USDA  published  a  proposed  rule  that  updated  and 
expanded  the  nutrition  standards  for  school  meals  and  required 
them  to  meet  the  DGA.  In  particular,  school  meals  were  to  satisfy 


®Burghardt,  et  al.,  1993. 

®60  FR  31195,  June  13, 1995. 
FR  31196,  June  13, 1995. 


4  State  Monitoring  of  NSLP  Nutritional  Content 


quantitative  standards  for  percentage  of  calories  from  fat,  percent¬ 
age  of  calories  from  saturated  fat,  protein  (g),  calcium  (mg),  iron 
(mg),  vitamin  A  (RE),  vitamin  C  (mg),  and  calories  (kcal).  In  addition, 
qualitative  goals  stated  that  school  meals  should  show  a  reduction  in 
the  level  of  cholesterol  and  sodium  and  an  increase  in  the  level  of 
fiber.  Since  the  rule  gives  no  quantitative  standards  with  which  to 
measure  these  changes,  states  were  to  be  allowed  to  achieve  these 
guidelines  in  one  of  two  ways.  They  could  choose  a  quantitative 
standard  for  these  nutrients  and  show  how  the  nutrient  content  of 
their  school  meals  compares  to  this  standard  or  they  could  choose  to 
use  an  SFA's  initial  nutrient  analysis  as  a  baseline  and  show  the 
change  in  subsequent  nutrient  analyses.  See  Table  1.1  for  the  cur¬ 
rent  standards  by  grade  range. 

To  assure  that  school  meals  served  satisfied  the  DGA,  the  proposed 
rule  would  have  replaced  the  NSLP's  rigid  food-based  menu  plan¬ 
ning  system  that  had  been  in  place  since  the  beginning  of  the  pro¬ 
gram  with  a  menu  planning  system  based  on  computerized  nutrient 
analysis.  In  particular,  the  Nutrient  Standard  Menu  Planning 
(NuMenus)  system  would  have  required  SFAs  to  perform  a  nutrient 
analysis,  using  USDA-approved  computer  software,  on  all  foods  of¬ 
fered  as  part  of  a  reimbursable  meal  in  the  meals  programs  prior  to 
the  service  of  the  meals. 


Table  1.1 

Nutrient  Standards — NuMenus  and  Assisted  NuMenus 


Standards  by  Grade  Range 

Nutrient 

Preschool 

K-6 

7-12 

Optional  K-3 

Calories  (kcal) 

>517 

>664 

>825 

>633 

Total  Fat  (as  %  of  food 

<30% 

<30% 

<30% 

<30% 

energy) 

Saturated  Fat  (as  %  of 

<10% 

<10% 

<10% 

>10% 

food  energy) 

Protein  (g) 

>7 

>10 

>16 

>9 

Calcium  (mg) 

>267 

>286 

>400 

>267 

Iron  (mg) 

>3.3 

>3.5 

>4.5 

>3.3 

Vitamin  A  (RE) 

>150 

>224 

>300 

>200 

Vitamin  C  (mg) 

>14 

>15 

>18 

>15 

Cholesterol  (mg) 

A  reduction  in  level  of  cholesterol 

Sodium  (mg) 

A  reduction  in  the  level  of  sodium 

Fiber  (g) 

An  increase  in  the  level  of  fiber 

Introduction  5 


The  computerized  nutrient  analysis  would  have  provided  the  SFA 
with  a  nutrient  analysis  of  each  proposed  menu.  Given  that  analysis, 
it  would  have  been  straightforward  to  verify  that  the  proposed  menu 
was  in  compliance  with  the  DGA,  and  thus  with  the  requirements  of 
the  NSLP  authorizing  legislation.  If  trial  menus  were  not  in  compli¬ 
ance,  SFAs  could  iteratively  make  adjustments  to  the  trial  menu  until 
it  was  in  compliance. 

The  proposed  rule  recognized  that  not  all  SFAs  would  have  the  ca¬ 
pability  to  do  nutrient  analysis  and  therefore  provided  for  a  variation 
on  this  system.  That  variation,  called  Assisted  NuMenus,  allowed  the 
SFA  to  have  a  third  party  perform  the  nutrient  analysis.  Finally,  the 
proposed  regulation  required  the  state  agencies  to  assess  the  nutri¬ 
ent  analyses  being  performed  by  the  SFAs  and  to  take  appropriate 
actions  if  either  the  analyses  were  not  being  performed  correctly  or 
the  meals  were  not  in  compliance  with  the  DGA. 

Thus,  through  NuMenus  or  Assisted  NuMenus,  the  proposed  rule 
included  a  mechanism  through  which  school  meals  could  plausibly 
be  expected  to  be  in  compliance  with  the  DGA  by  the  1998-1999 
school  year.  However,  before  the  rule  could  be  finalized,  PL  103-448, 
the  Healthy  Meals  for  Healthy  Americans  Act  of  1994,  was  passed. 
Responding  to  widespread  complaints  about  the  burden  of  nutrient 
analysis,  the  legislation  instructed  USDA  to  allow  SFAs  to  continue 
using  a  food-based  system  of  menu  planning,  in  addition  to  Nu¬ 
Menus  or  Assisted  NuMenus.  The  legislation  also  required  that 
school  meals  be  in  compliance  with  the  nutrient  requirements  by  the 
1996-1997  school  year  (two  years  earlier  than  under  the  proposed 
rule).  However,  since  the  legislation  allowed  menu  planning  systems 
that  did  not  automatically  involve  nutrient  analysis,  there  was  no  di¬ 
rect  mechanism  through  which  food-based  SFAs  could  be  expected 
to  verify  that  their  meals  were  indeed  in  compliance  with  the  DGA. 

On  January  27,  1995,  USDA  published  a  revised  proposed  rule.  It 
supplemented  the  June  10,  1994  proposed  rule.  It  incorporated  the 
shorter  timeline  and  the  inclusion  of  food-based  menu  planning 
systems  from  PL  103-448.  In  the  January  rule,  USDA  proposed  a 
food-based  system,  called  Enhanced  Food-Based  menu  planning, 
that  was  very  similar  to  the  previous  food-based  system.  The  only 
difference  from  a  traditional  food-based  system  was  the  inclusion  of 
more  fruits  and  vegetables  and  grains  and  breads.  To  implement  the 
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requirement  that  NLSP  meals  meet  the  DGA,  but  in  the  absence  of  a 
requirement  that  every  SFA  perform  a  computerized  nutrient  analy¬ 
sis  for  every  menu,  the  revised  proposed  regulation  required  each 
State  Agency  to  monitor  those  SFAs  using  food-based  systems.  That 
monitoring  was  to  consist  of  a  nutrient  analysis  of  each  SFA  at  least 
every  five  years. 

THE  FINAL  RULE  AND  THE  HEALTHY  MEALS  FOR 
CHILDREN  ACT 

The  final  rule,  which  was  published  June  13,  1995,  in  the  Federal 
Register  and  was  based  on  the  proposed  rules  from  June  1994  and 
January  1995.  In  May  1996,  Congress  enacted  the  Healthy  Meals  for 
Children  Act  (PL  104-149),  which  added  two  more  menu  planning 
systems — the  system  that  had  been  in  place  since  1946  (Traditional 
Food-Based)  and  another  called  “Any  Reasonable  Approach."  Under 
the  latter  option,  states  could  develop  their  own  menu  planning 
system  as  long  as  it  met  the  requirements  laid  out  by  FNS  for  school 
meals  and  nutrition. 

Since  the  final  regulations  allowed  for  five  different  menu  planning 
systems  (see  Table  1.2),  several  of  which  did  not  include  having  the 
SFA  perform  nutrient  analysis,  it  required  much  more  work  for  the 
State  Agency  than  the  June  1994  version  of  the  rule  would  have.  Un¬ 
der  the  original  proposed  rule,  all  SFAs  would  have  been  required  to 
perform  their  own  nutrient  analyses  or  hire  someone  to  do  the  analy¬ 
ses  because  they  would  have  had  to  use  NuMenus  or  Assisted 

Table  1.2 

Menu  Planning  Systems  in  the  NSLP 


Menu  Planning  System 

Description 

NuMenus 

Assisted  NuMenus 

Traditional  food-based 
menu  planning 

Enhanced  food-based 
menu  planning 

Any  reasonable  approach 

Nutrient  analysis  done  by  SFA 

Nutrient  analysis  done  by  third  party  (e.g.,  State 
Agency  or  outside  contractor) 

Similar  to  old  system;  no  nutrient  analysis  except 
during  SMI  review 

Similar  to  old  system  but  with  more  fruits,  veg¬ 
etables,  bread,  and  grains;  no  nutrient  analysis 
except  during  SMI  review 

Any  other  system  approved  by  USDA 
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NuMenus.  While  the  final  rule  and  the  mandates  under  the  Healthy 
Meals  for  Children  Act  allowed  the  SFAs  more  flexibility,  they  also 
gave  the  SFAs  the  opportunity  to  choose  menu  planning  systems  that 
did  not  require  them  to  perform  their  own  analyses.  Therefore,  the 
states  were  required  to  put  a  system  in  place  that  would  allow  state 
staff  or  contractors  to  perform  the  analyses. 

Under  the  current  SMI  regulations,  states  are  required  to  monitor  the 
nutritioned  content  of  school  meals,  but  they  are  not  required  to  for¬ 
ward  any  information  to  USDA — either  about  which  reviews  have 
been  completed  or  about  the  nutritional  content  of  the  meals  re¬ 
viewed  or  about  the  improvement  plans  developed.  FNS,  however,  is 
required  to  measure  progress  against  Objective  2.1  of  FNS's  1997- 
2002  Strategic  Plan,  which  states  that  FNS  will  “ensure  that  school 
meals  are  consistent  with  the  Dietary  Guidelines  for  Americans  and 
the  Recommended  Daily  Allowances."  The  “target"  established  un¬ 
der  Objective  1.3  is  “By  2005,  reach  less  than  or  equal  to  30%  calories 
from  total  fat  and  less  than  10%  calories  from  saturated  fat;  maintain 
calorie,  vitamin  and  mineral  content  at  greater  than  or  equal  to  33% 
of  RDA."  To  do  this,  FNS  needs  to  produce  state  and  national  aggre¬ 
gations  of  the  nutrient  content  in  school  lunches,  so  that  FNS  can 
show  that  meals  in  a  given  state  or  in  the  country  as  a  whole  are 
meeting  the  given  objective  of  ensuring  that  school  meals  are  consis¬ 
tent  with  the  Dietary 

Guidelines  and  the  RDAs.  In  the  short-term,  USDA  desires  “to  iden¬ 
tify  potential  reporting  systems  that  could  obtain  for  FNS  the  data 
necessary  to  measure  the  nutrient  content  of  school  lunches  against 
the  nutrient  standards  in  Objective  2.1  of  FNS’s  strategic  plan.”^^ 
Specifically,  FNS  staff  “need  to  aggregate  nutrient  analysis  data  from 
all  SMI  reviews  in  a  state  so  that  the  aggregated  data  represents  the 
nutrient  content  of  all  reviewed  meals  in  the  state  [and]  aggregate 
nutrient  analysis  data  from  all  SMI  reviews  nationwide  so  that  the 
aggregated  data  represents  the  nutrient  content  of  all  reviewed  meals 

nationwide.  ”12 


^Statement  of  Work,  Child  Nutrition  Analysis  Projects,  Task  Order  #4. 

l^ibid.  Note  that  this  Statement  of  Work  refers  to  “reviewed  meals”  while  the  Strategic 
Plan  refers  to  school  meals  in  general.  See  Chapter  5  for  a  discussion  on  “meals 
reviewed”  versus  “meals  served"  and  statistical  sampling. 
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OBJECTIVES  AND  APPROACH 

To  explore  issues  related  to  state  reporting  of  the  results  of  SFA-level 
SMI  review,  USDA  contracted  with  RAND  to  conduct  a  closed- 
response  telephone  survey  of  all  50  states.  After  the  original  RAND 
interview  protocol  was  developed  and  reviewed  by  the  Education  In¬ 
formation  Advisory  Committee  (ELAC),  USDA  redesigned  the  task  to 
involve  in-person,  in-depth,  interviews  with  State  Child  Nutrition  Di¬ 
rectors  in  seven  volunteer  states.  This  change  was  made  to  respond 
to  concerns  of  committee  members  and  other  State  Child  Nutrition 
Directors  that  FNS  was  performing  the  survey  as  a  precursor  to  man¬ 
dating  a  nationwide  reporting  system  for  SMI  data.  Rather  than 
mandate  anything,  FNS  decided  to  look  more  carefully  at  how  some 
states  are  performing  SMI  reviews  and  how  they  might  be  able  to 
transmit  data  with  limited  additional  burden. 

FNS  asked  seven  states  to  work  collaboratively  with  RAND — through 
on-site,  in-depth  interviews — to  explore  possible  reporting  systems. 
FNS  chose  the  seven  states  to  be  used  for  the  case  studies.  Figure  1.1 
shows  the  seven  states,  which  were  chosen  to  represent  very  differ¬ 
ent  methods  for  performing  SMI  reviews  and  the  seven  FNS  regions. 
The  seven  volunteer  states  are  shaded  and  the  thick  black  lines  out¬ 
line  the  regions  they  represent. 


RANDMR 1296- 1.1 


Figure  1.1 — Seven  Volunteer  States  Selected  and  the  FNS  Regions 
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At  the  beginning  of  1999,  the  RAND  team  contacted  the  State  Child 
Nutrition  Directors  or  other  main  contact  in  each  of  these  seven 
states  by  telephone  to  schedule  the  interviews.  The  interview  proto¬ 
cols  for  the  site  visits  were  based  on  the  interview  protocol  developed 
for  the  original  telephone  survey,  with  additional  probes  to  obtain 
specific  details  on  how  the  SMI  review  process  worked  in  that  state 
and  the  likely  impact  of  a  data  request  from  FNS.  After  the  initial 
telephone  contact,  copies  of  the  interview  protocol  were  faxed  to  the 
states.  State  contacts  were  asked  to  complete  a  short  background 
questionnaire  and  return  it  to  RAND  before  the  scheduled  meeting. 
The  short  background  questionnaire  included  questions  about  the 
number  of  SFAs  in  the  state  and  the  number  of  SMI  reviews  com¬ 
pleted.  One  or  two  RAND  staff  members  visited  each  of  the  states  for 
an  in-depth  interview.  After  the  interview,  the  RAND  team  compiled 
the  state  official  answers  to  the  interview  questions  and  sent  the 
written  answers  back  to  the  states  for  comments,  giving  them  the 
opportunity  to  correct  any  misinformation  or  add  any  missing  in¬ 
formation. 

INDIVIDUALS  SELECTED  FOR  INTERVIEWS  IN  THE  SEVEN 
STATES 

In  each  state  except  California  and  Wisconsin,  the  State  Agency  Di¬ 
rector  was  contacted  and  that  person  assisted  us  in  arranging  to 
speak  to  other  key  people.  In  California  and  Wisconsin,  the  original 
contact  person  was  someone  within  the  Agency,  though  not  the  di¬ 
rector.  Again,  the  contact  person  arranged  a  meeting  at  which  time 
RAND  spoke  to  the  contact  and  to  other  important  individuals  in  the 
state.  See  Appendix  B  for  a  complete  list  of  the  individuals  inter¬ 
viewed  in  each  of  the  seven  states. 

ORGANIZATION  OF  THIS  DOCUMENT 

This  document  describes  the  results  of  those  site  visits  and  their 
implications.  Chapter  2  discusses  what  data  elements  should  be  sent 
to  FNS  to  make  analysis  possible,  which  elements  could  be  useful  but 
are  not  necessary,  and  which  elements  we  recommend  against  col¬ 
lecting.  Chapter  3  describes  SMI  reviews  in  general  terms  and  how 
they  are  conducted  in  the  seven  volunteer  states.  Chapter  4  dis¬ 
cusses  how  the  data  elements  should  be  sent  to  FNS.  Chapter  5  dis- 


10  State  Monitoring  of  NSLP  Nutritional  Content 


cusses  some  broader  issues  in  the  design  of  the  SMI  review  effort. 
The  complete  interview  protocol  can  be  found  in  Appendix  A  of  this 
document.  The  completed  responses  to  the  interview  protocol  for  all 
seven  states  can  be  found  in  an  appendix  volume. 


Chapter  Two 

WHAT  DATA  SHOULD  BE  PROVIDED? 


Although  the  states  are  required  to  conduct  SMI  reviews,  USDA-FNS 
currently  receives  no  information  on  the  results  of  the  reviews  or 
even  which  reviews  have  been  conducted.  The  main  purpose  of  this 
study  was  to  explore  approaches  that  USDA  could  use  to  collect  in¬ 
formation  on  the  results  of  SFA-level  SMI  reviews,  while  imposing 
minimal  burden  on  the  states  and  the  SFAs,  To  design  an  appropri¬ 
ate  system,  USDA  needs  to  determine  what  information  should  be 
reported  and  how  that  information  should  reach  USDA.  We  discuss 
“what”  in  this  chapter  and  “how”  in  Chapter  4. 

What  should  be  collected  depends  on  the  uses  USDA  plans  to  make 
of  the  information  it  receives.  To  assess  progress  toward  the  items  in 
its  strategic  plan,  USDA  would  like  to  be  able  to  tabulate  the  share  of 
meals  served  in  reviewed  schools  that  meet  each  of  the  guidelines 
(and  combinations  of  the  guidelines).  Furthermore,  USDA  would 
like  to  be  able  to  do  the  tabulations  by  state,  type  of  institution,  and 
age /grade  range. 

Among  the  items  we  discussed  with  State  Agency  staff  during  the  site 
visits  was  a  list  of  data  elements  to  be  collected  (see  the  complete 
interview  protocol  in  Appendix  A),  FNS  provided  the  initial  list. 
During  the  discussion  with  the  states  and  during  our  own  analysis, 
we  reconsidered  the  list  of  elements  to  be  collected.  The  list  in¬ 
cluded  in  Appendix  A  is  an  exhaustive  list  of  potential  data  elements 
to  collect.  However,  it  appears  that  not  all  of  these  elements  are  nec¬ 
essary  and  some  of  them  probably  should  not  be  collected  at  all. 

We  begin  by  describing  the  data  elements  that  we  recommend  that 
USDA  should  collect,  along  with  some  that  might  be  useful  but  that 
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are  not  necessary  for  analysis.  We  then  describe  the  elements  that 
were  considered  for  inclusion  (either  because  they  were  suggested  by 
USDA-FNS  or  by  a  state  during  an  interview),  but  that  we  recom¬ 
mend  not  be  included. 

REQUIRED  ELEMENTS 

The  following  screen  graphically  depicts  the  list  of  data  elements  that 
we  have  concluded  USDA  needs  to  perform  its  desired  tabulations. 

Type  of  SFA:  An  indicator  for  public  school  or  school  district,  private 
school  or  school  district,  residential  child  care  institution  (RCCI),  or 
other  type  of  SFA  is  important  because  different  types  of  SFAs  may  be 
better  prepared  for  SMI  reviews  than  others.  All  the  states  to  which 
we  spoke  indicated  that  RCCIs  are  not  ready  for  SMI  because  of  lack 
of  staff.  In  addition,  FNS  has  said  that  it  is  currently  focusing  its 
resources  helping  SFAs  meet  SMI  requirements.  Therefore,  FNS  will 
need  to  know  whether  results  are  for  an  RCCI  or  not.  Although  not 
necessarily  collected  as  part  of  the  current  SMI  review,  this  data  ele¬ 
ment  should  be  very  easy  for  the  states  to  collect  in  the  future.  The 
software  should  include  a  menu  of  choices. 


Type  of  SFA 

Answer  Code 

Public  School  or  District 

1 

Private  School  or  District 

2 

RCCI 

3 

Other 

9 

Menu  Planning  System  Used:  An  indicator  for  traditional  food- 
based  menu  planning,  enhanced  food-based  menu  planning,  nutri¬ 
ent  standard  menu  planning,  assisted  nutrient  standard  menu  plan¬ 
ning,  or  “any  reasonable  approach”  is  needed.  Depending  on  the 
type  of  menu  planning  system  used,  the  SFA  may  do  the  nutrient 
analysis  or  the  state  may  do  it.  Knowing  who  did  the  analysis  will  be 
useful  if  FNS  sees  consistent  problems  with  either  the  state’s  man¬ 
agement  of  analysis  or  the  way  schools  are  doing  analysis.  Also, 
states  have  to  be  aware  of  the  menu  planning  system  used  in  order  to 
proceed  with  the  SMI  review.  Therefore,  this  information  should  be 
readily  available  during  the  review  and  will  only  need  to  be  added  to 
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the  information  recorded  as  part  of  the  review.  The  software  should 
include  a  menu  of  choices. 


Type  of  Menu  Plan 

Answer  Code 

Nutrient  Standard  Menu  Planning 

1 

Assisted  Nutrient  Standard  Menu  Planning 

2 

Enhanced  Food-Based 

3 

Traditional  Food-Based 

4 

Any  Reasonable  Approach 

9 

Analysis  Lunch  Only,  Breakfast  Only,  or  Lunch  and  Breakfast:  FNS 
is  interested  only  in  analyses  of  lunch  menus  for  this  project.  How¬ 
ever,  some  reviews  and  analyses  are  completed  for  both  lunch  and 
breakfast.  Therefore,  FNS  needs  to  know  which  meals  are  covered  by 
an  analysis.  This  information  will  be  provided  by  the  SFA.  As  with 
the  type  of  SFA  and  the  type  of  menu  plan,  it  will  merely  need  to  be 
added  to  the  information  recorded.  The  software  should  include  a 
menu  of  choices. 


Meal  Analyzed 

Answer  Code 

Lunch  Only 

1 

Breakfast  Only 

2 

Lunch  and  Breakfast 

3 

Analysis  Weighted:  Some  analyses  are  weighted,  either  by  type  of 
entree  served  or  by  all  menu  items — in  this  case,  weighting  refers  to 
allowing  for  differences  depending  on  the  number  of  portions  of  a 
particular  entree  or  other  item  that  are  served.  If  pizza  and  chicken 
are  both  served,  but  pizza  is  served  to  75  kids  and  chicken  to  only  25, 
then  the  pizza  would  be  weighted  75  percent  and  the  chicken  only  25 
percent  in  the  analysis.  Knowing  whether  an  analysis  was  weighted 
or  not  allows  USDA  to  make  comparisons  among  similar  strategies. 
For  SFAs  using  NuMenus  or  Assisted  NuMenus  in  states  where 
weighting  is  not  mandated,  the  state  will  have  to  find  out  from  the 
SFA  whether  weighting  was  used  or  not.  For  the  SFAs  using  other 
menu  planning  systems,  the  state  will  do  the  analysis  itself  and  will 
know  whether  weighting  was  used.  The  software  should  include  a 
menu  of  choices. 
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Analysis  Weighted 

Answer  Code 

Yes,  entree  only 

1 

Yes,  all  menu  items 

2 

No 

3 

Review  Done  by  Grade  or  Age:  An  indicator  of  grade  or  age  review  is 
needed  to  interpret  the  next  two  elements.  This  information  is 
needed  to  apply  the  appropriate  nutrient  standards  and  so  will  be 
readily  available.  The  software  should  include  a  menu  of  choices. 


Review  Type 

Answer  Code 

Age  Range 

1 

Grade  Range 

2 

Lowest  Grade/Age:  Lowest  age  or  grade  range  for  the  analysis  being 
reported  is  important  to  assess  which  standards  should  be  used  and 
to  allow  for  comparisons  by  age  or  grade  range.  Some  states  are  using 
standardized  age/grade  ranges  and  others  are  using  customized 


Lowest  Grade /Age 
P 
K 
1 
2 

3 

4 

5 

6 

7 

8 

9 

10 
11 
12 

13 

14 

15 

16 

17 

18 


Answer  Code 

19 

20 
1 
2 

3 

4 

5 

6 

7 

8 
9 

10 

11 

12 

13 

14 

15 

16 

17 

18 
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ones.  This  will  allow  USDA  to  see  which  is  being  used.  This  informa¬ 
tion  should  already  be  available  since  the  computer  software  re¬ 
quires  it  to  determine  the  standards  against  which  the  SFA  data 
should  be  compared.  A  menu  of  answer  choices  should  be  included. 

Highest  Grade/Age:  Highest  age  or  grade  range  for  the  analysis  be¬ 
ing  reported  is  important  in  order  to  assess  which  standards  should 
be  used  and  to  allow  for  comparisons  by  age  or  grade  range.  This 
information  should  already  be  available  since  the  computer  software 
requires  it  to  determine  the  standards  against  which  the  SFA  data 
should  be  compared.  The  menu  of  answer  choices  would  be  identi¬ 
cal  to  the  one  for  lowest  age/grade  range. 

Estimated  Average  Daily  Participation:  This  number  is  the  esti¬ 
mated  average  daily  school  lunch  participation  during  the  SMI  re¬ 
view  week  in  the  school  that  administered  the  SMI  review  for  the 
student  age/grade  range  covered  by  this  analysis.  This  figure  can  be 
used  as  a  weight  so  that  nutrient  analysis  results  from  different 
schools  or  different  states  can  be  aggregated.  With  this  weighting  el¬ 
ement,  FNS  will  be  able  to  calculate  the  nutritional  content  of  the 
meals  that  the  average  child  in  the  reviewed  schools  is  getting,  i.e., 
larger  schools  get  larger  weights.^ 

This  number  may  not  be  readily  available.  Though  SFAs  have  aggre¬ 
gate  data  on  meals  served,  they  may  have  to  estimate  the  number  of 
meals  served  for  a  given  age  or  grade  range  represented  by  the  anal¬ 
ysis.  This  information  can  be  derived  from  the  same  sources  used  to 
report  the  number  of  meals  served  in  the  Coordinated  Review  Effort. 
Given  the  type  of  analysis  that  FNS  will  be  able  to  do  with  these  data, 
an  exact  number  is  probably  not  critical.  Instead,  an  estimate  by  the 
SFA  seems  sufficient.  This  is  the  one  required  data  element  that  may 
cause  a  burden  to  some  of  the  states. 

Actual  Output  from  Analysis  Software:  These  numbers  are  the  data 
on  the  eleven  items  that  the  USDA  requires — calories,  fat,  saturated 
fat,  protein,  calcium,  iron,  vitamin  A,  vitamin  C,  sodium,  cholesterol, 


^  See  the  discussion  in  Chapter  5  about  whether  we  want  to  represent  meals  reviewed 
or  meals  served. 
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and  fiber.2  The  software  provides  quantitative  measures  of  all  eleven 
elements.  This  information  would  come  directly  from  the  output  of 
the  nutrient  analysis. 

OPTIONAL  DATA  ELEMENTS 

SFA  Identifier:  A  unique  identifier  for  each  SFA  is  needed  to  distin¬ 
guish  the  records  in  the  database.  The  SFA  agreement  number  is  the 
natural  candidate,  but  states  are  somewhat  anxious  that  identifying 
an  SFA  in  any  way,  even  with  a  random  identifier,  may  allow  FNS  to 
find  and  punish  SFAs  with  poor  results  on  nutrient  analyses.  To 
eliminate  this  possibility,  the  FNS  software  could  automatically  as¬ 
sign  a  numeric  ID  number  (e.g.,  the  sequence  number).  If  states 
wanted  to  provide  this,  they  could  use  the  SFA  identifier  that  they  al¬ 
ready  have  in  place  with  a  year  appended  in  order  to  make  it  unique 
for  each  analysis  or  they  could  create  an  ID  number  in  numeric  order 
as  the  reviews  are  completed.  If  a  state  decides  to  include  this,  the 
burden  would  be  minimal. 

Review  Week:  The  dates  for  the  week  in  which  menus  were  reviewed 
for  the  nutrient  analysis  may  be  useful  to  see  any  seasonality  effect 
on  the  analysis.  For  example,  if  an  analysis  is  done  the  week  of 
Thanksgiving,  the  school  may  serve  a  special  holiday  meal,  which  is 
high  in  fat.  This  data  element  should  not  be  difficult  to  get  because  it 
should  be  in  the  NutriKids^  output.  However,  because  of  peculiari¬ 
ties  of  NutriKids,  if  too  many  reviews  happen  on  the  same  week,  the 
reviewers  may  have  to  change  the  dates  when  entering  them  since 
NutriKids  can  handle  only  a  limited  number  of  reviews  in  a  given 
week.  Therefore,  even  dates  of  the  review  week  may  be  somewhat 
difficult. 

Contact  Person:  The  name  of  someone  FNS  can  call  with  any  ques¬ 
tions.  This  contact  could  be  the  same  person  for  all  analyses  from 


^  While  FNS  does  not  have  quantitative  measures  for  sodium,  cholesterol,  and  fiber, 
the  regulations  say  that  the  states  should  show  a  decrease  in  sodium  and  cholesterol 
and  an  increase  in  fiber.  This  regulation  can  be  met  by  creating  quantitative  measures 
for  these  items  within  the  state  or  by  showing  changes  from  one  SMI  cycle  to  the  next. 

^  We  have  specifically  used  NutriKids  in  our  text  because  everyone  we  spoke  to  was 
using  NutriKids.  Other  software  packages  are  acceptable  according  to  FNS,  but  we 
have  not  addressed  them  because  we  heard  nothing  about  them  in  our  interviews. 
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each  state  or  could  be  different  people  based  on  who  did  the  analy¬ 
sis.  In  either  case,  this  should  be  relatively  easy  for  states  to  provide, 
should  they  decide  to  do  so.  The  states  suggested  that  the  contact 
person  be  the  State  Director.  This  data  element  could  be  optional 
because  FNS  indicates  that  it  does  not  currently  have  the  staff  for 
going  back  to  respondents  to  clarify  information.  If  there  were  ques¬ 
tions  that  absolutely  had  to  be  answered,  FNS  would  go  back  to  the 
State  Director  for  clarification.  The  State  Directors  also  prefer  this 
approach. 

Clarification  of  “Any  Reasonable  Approach”:  A  brief  description  of 
any  nonstandard  menu  planning  system  used  could  be  optional  be¬ 
cause  few  SFAs  use  “any  reasonable  approach”  as  a  menu  planning 
system.  In  addition,  across  SFAs  that  use  “any  reasonable  ap¬ 
proach,”  there  is  little  consistency.  However,  we  include  it  as  an  op¬ 
tional  data  element  because  it  may  be  useful  to  know  the  type  of 
system  used,  especicdly  if  the  results  from  the  analysis  seem  different 
from  other  analyses  in  the  same  state  or  in  schools  of  similar  size  and 
type.  Even  if  the  descriptions  do  not  have  any  consistency  across 
SFAs,  they  could  be  useful  on  an  individual  basis.  Since  SFAs  have 
developed  their  own  systems  in  this  case,  giving  clarification  on  the 
approach  they  use  should  be  very  straightforward. 

Urban  or  Rural  Area;  A  designation  for  urban  or  rural  area  is  proba¬ 
bly  best  done  with  U.S.  Census  Bureau  designations,  where  an  urban 
area  comprises  “all  territory,  population,  and  housing  units  in  urban¬ 
ized  areas  and  in  places  of  2,500  or  more  persons  outside  urbanized 
areas. This  data  element  would  allow  FNS  to  differentiate  between 
large  schools  in  urban  and  rural  areas  since,  as  several  states  pointed 
out,  schools  in  rural  areas  may  be  less  well  prepared  for  SMI  reviews. 
Most  SFAs  will  probably  know  whether  the  area  they  are  located  in  is 
designated  urban  or  rural.  If  not,  getting  this  information  from  the 
U.S.  Census  Bureau  is  easy.  The  designation  would  just  need  to  be 
added  to  the  information  sent  to  the  state. 

Size  of  District:  A  measure  of  the  size  of  the  district — using  either 
enrollment  or  number  of  schools.  Enrollment  would  be  better  since 
schools  can  vary  widely  in  size.  However,  number  of  schools  could 


^From  the  U.S.  Census  Bureau  web  page  at  http://www.census.gov/population/ 
censusdata/urdef.txt. 
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be  useful  if  no  enrollment  figures  were  available.  In  either  case,  the 
SFA  would  have  to  provide  this  number.  Most  SFAs  will  probably 
know  enrollment  figures  and  should  be  able  to  provide  them  readily 
to  the  state.  This  data  element  would  be  useful  for  comparing  dis¬ 
tricts.  As  several  states  pointed  out,  it  would  not  be  fair  to  compare 
districts  with  100  kids  to  those  with  100,000  kids. 

Comments:  Any  additional  comments  or  clarifications  that  the  state 
wishes  to  meike  on  the  analysis  on  which  they  are  reporting  could  be 
input  here.  Including  this  field  gives  the  states  a  place  to  document 
any  strange  circumstances  surrounding  a  particular  review. 

DATA  ELEMENTS  THAT  SHOULD  NOT  BE  INCLUDED 

Several  other  data  elements  were  suggested  either  by  FNS  or  by  the 
states.  We  recommend  against  requiring  that  they  be  reported. 
These  elements  include  the  name  of  the  person  performing  the  anal¬ 
ysis,  the  date  the  review  was  closed,  and  the  standards  with  which 
the  quantitative  elements  are  being  compared. 

Name  of  Reviewer:  The  states  were  very  strongly  opposed  to  provid¬ 
ing  the  names  of  reviewers,  consultants,  or  subcontractors.  The  State 
Directors  consistently  stated  that  since  it  was  their  responsibility  to 
oversee  the  SMI  process,  FNS  should  contact  them  with  any  ques¬ 
tions  once  the  data  are  submitted.  Additionally,  if  FNS  examines  the 
data  annually,  some  of  the  reviews  would  be  a  year  or  more  old. 
Thus,  even  if  FNS  had  questions,  it  is  likely  that  the  states  would  no 
longer  have  the  answers,  so  a  contact  person  would  be  irrelevant. 

Dates  of  Review:  The  states  reported  that  providing  an  ending  date 
for  a  review  could  be  very  difficult,  though  they  could  probably  pro¬ 
vide  a  start.  It  is  often  hard  to  determine  when  the  review  is  com¬ 
plete.  A  review  may  be  complete  when  a  correction  plan  is  imple¬ 
mented,  when  a  new  analysis  is  run,  when  a  closeout  letter  is  sent, 
depending  on  the  state. 

Standards:  USDA  suggested  requiring  the  states  to  provide  the  stan¬ 
dards  against  which  the  data  from  nutrient  analysis  are  compared. 
The  states,  however,  noted  that  in  most  cases  USDA  mandates  the 
standards.  Thus,  knowledge  of  the  type  of  menu  planning  system 
used  and  the  age  or  grade  range  is  usually  enough  to  identify  the 
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applicable  standard.  The  exception  is  states  that  are  using  a  menu 
planning  system  under  “any  reasonable  approach”  where  FNS  will 
not  have  the  standards.  In  this  case,  it  would  be  sufficient  for  the 
state  to  submit  the  standards  to  USDA  annually  along  with,  but  sepa¬ 
rate  from,  the  nutrient  analysis  results.  It  seems  unnecessary  and 
potentially  error  prone  to  require  the  states  to  resubmit  the 
standards  with  each  analysis.  One  possibility  would  be  for  USDA’s 
program  to  pre-fill  the  standards  into  the  worksheet.  Since  NutriKids 
can  automatically  assign  the  standards  given  the  age /grade  range,  it 
should  not  be  too  difficult  to  allow  this  to  happen  in  other  software. 


Chapter  Three 

CURRENT  REVIEW  PROCEDURES 


USDA  desires  to  understand  each  state’s  ability  to  report  the  re¬ 
quired  information  and  to  develop  a  system  that  would  impose  the 
minimal  burden  on  the  states.  To  do  so,  it  is  crucial  that  USDA  un¬ 
derstand  the  systems  that  states  are  currently  using  to  perform  their 
SMI  reviews  and,  in  particular,  the  flow  of  the  required  information 
from  the  states  to  USDA  from  the  time  the  nutritional  analysis  is 
completed.  This  chapter  includes  this  information  for  the  seven  vol¬ 
unteer  states.  The  chapter  first  describes  the  general  SMI  review 
process  and  then  summarizes  the  process  in  each  of  the  seven  states. 

OVERVIEW  OF  SMI  REVIEW  PROCESS  IN  THE  SEVEN 
STATES 

When  we  interviewed  individuals  within  the  states  about  the  process 
they  use  to  conduct  SMI  reviews,  we  discovered  some  similarities. 
Figure  3.1  shows  the  basic  steps  for  completing  an  SMI  review  across 
the  seven  states.  The  solid  arrows  indicate  steps  that  every  state  fol¬ 
lows.  The  dashed  lines  indicate  steps  that  may  occur  in  some  states 
or  for  some  analyses,  but  that  are  not  always  present.  This  figure 
does  not  include  every  step  that  every  state  takes  in  completing  SMI 
reviews.  Instead,  it  gives  a  basic  outline  of  the  steps  that  most  states 
use. 

In  every  state,  the  SFA  provides  specific  background  information  for 
the  review  week.  If  the  SFA  is  not  using  a  nutrient-based  menu 
planning  system,  this  information  includes  menus,  recipes,  and  pro¬ 
duction  records  to  the  state  or  a  contractor.  For  SFAs  using  nutrient- 
based  plans,  they  also  provide  the  nutrient  analysis  for  review.  In 
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Figure  3.1 — Basic  Steps  in  Completing  the  SMI  Review  Across  States 


some  cases,  this  information  is  sent  prior  to  the  SMI  site  visit,  and  in 
other  cases  it  is  collected  during  the  site  visit;  but  the  information  is 
always  required  from  the  SFA.  Sometimes  the  SFA  does  not  have  ev¬ 
erything  available  prior  to  the  visit,  and  it  requires  several  weeks  or 
even  months  after  the  visit  to  collect  all  the  necessary  information. 
Once  the  background  information  is  collected,  it  is  sent  to  the  person 
responsible  for  completing  the  nutrient  analysis.  In  some  cases,  the 
same  person  who  collected  the  information  performs  the  nutrient 
analysis.  In  other  cases,  the  information  is  sent  to  another  person — 
either  within  the  state  agency  or  at  an  outside  contractor — for  the 
analysis.  When  the  analysis  is  complete,  a  correction  plan  is  devel¬ 
oped  with  the  SFA,  and  the  review  is  closed. 

If  an  SFA  is  using  a  nutrient-based  menu  planning  system,  often  the 
state  authority  merely  collects  the  output  of  the  analysis  program. 
The  analysis  is  usually  not  redone  by  the  state.  Instead,  the  reviewer 
examines  the  completed  analysis  sent  by  the  SFA. 

HOW  THE  PROCESS  WORKS  WITHIN  EACH  OF  THE  SEVEN 
STATES 

Each  state  operates  slightly  differently,  and,  even  within  a  given  state, 
reviewers  may  alter  this  process  somewhat.  To  understand  that 
variation  across  states,  we  present  case  studies  for  seven  states  based 
on  our  site  visits.  These  case  studies  consider  the  following  issues: 
who  collects  background  information,  who  performs  analyses,  when 
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analyses  are  sent  to  the  state,  how  and  where  information  on  analy¬ 
ses  is  kept,  and  how  difficult  it  would  be  to  send  information  to  FNS. 

California 

In  California,  reviewers  collect  hard  copies  of  the  menus,  recipes, 
and  other  information  from  the  SFAs  during  the  site  visit.  The  re¬ 
viewers  are  located  throughout  the  state,  and  all  work  for  the  Cali¬ 
fornia  Department  of  Education.  In  most  cases,  the  reviewers  send 
the  information  to  the  Department  of  Education,  Nutrition  Stan¬ 
dards  Unit,  where  data  entry  staff  enter  it  into  the  NutriKids  pro¬ 
gram,  but  reviewers  will  occasionally  do  the  data  entry  and  perform 
the  nutrient  analyses  themselves.  Even  when  a  reviewer  does  the 
analysis,  it  is  reviewed  again  by  the  Nutrition  Standards  Unit.  If  the 
data  entry  staff  find  any  information  missing,  they  may  consult  with 
the  reviewers  or  follow  up  with  the  SFA  themselves. 

Analyses  that  are  completed  at  the  state  office  are  not  revised.  Analy¬ 
ses  done  by  the  reviewers  may  be  revised  if  the  Nutrition  Standards 
Unit  personnel  find  problems,  but  this  happens  only  rarely.  If  an 
SFA  uses  NuMenus  or  Assisted  NuMenus,  the  reviewers  will  examine 
the  analysis,  but  they  will  not  redo  it.  Once  the  analyses  are  finished, 
they  are  kept  at  the  state  office  both  in  hard  copy  and  in  electronic 
format.  California  has  started  logging  the  results  of  the  nutrient 
analyses  into  a  Microsoft  Access  database  with  the  1998-1999  re¬ 
views.  Given  that  the  state  already  has  a  database  set  up  and  has 
agreed  to  revise  the  database  to  include  whichever  data  elements 
FNS  requires,  it  would  be  a  relatively  easy  task  for  California  to  report 
nutrient  analysis  data. 

Once  an  analysis  is  completed,  an  improvement  plan  is  developed  in 
conjunction  with  the  SFA.  Even  if  the  SFA  meets  all  the  nutritional 
requirements,  there  is  usually  some  sort  of  improvement  plan.  When 
the  improvement  plan  is  accepted  by  the  SFA,  the  review  is  consid¬ 
ered  closed. 

There  are  1,496  SFAs  in  the  state  of  California.  As  of  June  1999,  206 
(14  percent)  SMI  reviews  had  been  completed  and  recorded  at  the 
state  office.  However,  the  actual  number  of  reviews  completed  in 
June  1999  was  307  (21  percent),  but  some  reviews  had  not  yet  been 
recorded  when  we  talked  to  the  California  staff.  The  SFAs  in  Califor- 
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nia  are  using  a  variety  of  menu  planning  systems.  About  20  percent 
of  SFAs  use  nutrient-based  plans,  more  than  35  percent  use  en¬ 
hanced  food-based  plans,  20  percent  use  traditional  food-based 
plans,  and  25  percent  use  “any  reasonable  approach.”  Although 
USDA  no  longer  requires  that  analyses  be  weighted,  some  California 
analyses  continue  to  be  weighted. 

Georgia 

In  Georgia,  state  employees,  called  consultants,  perform  the  SMI  re¬ 
views  with  the  help  of  secretaries.  (The  title  consultant  is  used  in 
Georgia,  Nebraska,  and  Wisconsin  for  reviewers  who  work  for  the 
State  Agency).  These  employees  work  for  the  Georgia  Department  of 
Education,  but  they  have  offices  throughout  the  state  to  ease  the 
performance  of  the  site  visits.  In  the  future,  the  state  would  like  to 
hire  a  subcontractor  to  help  with  nutrient  analysis.  According  to  the 
state  staff,  there  is  not  currently  budget  to  do  so.  The  consultants 
and  secretaries  are  responsible  for  collecting  all  raw  information 
from  the  SFAs  during  the  site  visits.  The  secretaries  enter  the  data 
into  NutriKids,  and  the  consultants  do  the  analyses. 

Once  the  analysis  is  completed,  an  improvement  plan  is  formulated 
and  the  SFA  changes  its  menus  accordingly.  The  analysis  is  then  re¬ 
done  based  on  these  changes.  About  50  percent  of  SFAs  require  a 
second  analysis  and  second  site  visit.  Once  the  improvement  plan  is 
implemented  and  the  analysis  is  redone,  the  review  is  complete. 

Georgia  does  not  currently  have  any  state  database  that  records  the 
results  of  the  nutrient  analyses,  and  funds  are  not  available  to  create 
such  a  database.  Hard  copies  of  the  analyses  are  kept  at  the  state  of¬ 
fices  and  are  not  maintained  in  the  field  offices.  No  soft  (computer 
readable)  copies  of  analyses  are  currently  kept  on  file.  Currently,  it 
would  be  a  fair  amount  of  work  for  Georgia  to  devise  a  method  for 
reporting  data  to  FNS.  They  do  not  have  any  system  in  place  to  enter 
the  data  from  the  nutrient  analyses  into  a  database  of  any  sort.  Ev¬ 
erything  would  have  to  be  keypunched  from  the  hard  copies  unless 
there  was  a  way  to  save  the  information  directly  from  NutriKids. 

The  Georgia  Department  of  Education  is  responsible  only  for  SMI 
reviews  in  public  SFAs,  so  it  must  review  180  SFAs.  As  of  July  1999, 
when  we  talked  to  the  state  staff,  they  had  completed  reviews  on  78 
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SFAs  (43  percent).  SFAs  in  Georgia  are  almost  all  (95  percent)  using 
traditional  food-based  menu  plans.  Georgia  requires  analyses  to  be 
weighted  by  type  of  entree  served. 

Nebraska 

Unlike  the  other  six  states,  Nebraska  performs  the  nutrient  analysis 
prior  to  the  site  visit  of  the  SFA.  Several  months  before  the  visit,  a 
consultant  who  works  for  the  State  Agency  calls  the  SFA  and  requests 
all  the  necessary  information.  If  the  SFA  uses  a  food-based  system,  it 
sends  the  menus,  recipes,  etc.,  to  an  independent  contractor  to  do 
the  analysis  and  the  contractor  forwards  the  completed  review  to  the 
consultant.  If  the  SFA  uses  a  nutrient-based  system,  it  sends  the 
analysis  itself  to  the  consultant  for  review. 

The  consultant  then  brings  the  completed  nutrient  analysis  to  the 
site  visit.  Since  the  nutrient  analysis  is  already  complete,  at  the  visit 
the  consultant  can  then  give  comments  to  the  SFA  and  ask  for  an  im¬ 
provement  plan. 

Once  the  state  reviews  the  information  from  the  analysis,  either  by 
the  contractor  or  directly  from  the  SFA,  it  is  logged  into  a  Microsoft 
Access  database.  Hard  copies  of  the  analyses  are  also  kept  at  the 
state  office.  The  electronic  copies  of  the  analyses  are  not  kept  past 
the  end  of  the  school  year  in  which  they  were  done.  Since  the  state 
already  has  a  Microsoft  Access  database  that  could  easily  be  modi¬ 
fied  to  fit  the  needs  of  FNS,  Nebraska  should  not  have  a  problem 
supplying  the  necessary  data  in  the  future,  assuming  it  can  be  done 
directly  from  the  database. 

Nebraska  has  506  SFAs  in  the  state  and  had  completed  reviews  on 
103  (20  percent)  as  of  July  1999.  SFAs  in  Nebraska  are  mostly  using 
food-based  menu  planning  systems,  with  about  80  percent  using  en¬ 
hanced  food-based  and  16  percent  using  traditional  food-based.  The 
rest  are  using  nutrient-based  systems.  Nebraska  requires  analyses  to 
be  weighted  by  type  of  entree. 

New  Jersey 

In  New  Jersey,  field  staff,  who  work  for  the  New  Jersey  Department  of 
Agriculture,  do  all  site  visits.  They  hand- carry  information  from  the 
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SFA  back  to  the  state  office  and  then  send  it  to  an  outside  contractor 
(located  in  Ohio)  for  the  nutrient  analysis.  The  contractor  sends  the 
completed  analyses  back  to  the  state  office,  and  the  reviewer  has  an 
exit  conference  with  the  SFA  to  discuss  improvement  plans.  The  re¬ 
viewer  then  sends  a  letter  to  the  district  outlining  an  improvement 
plan.  Finally,  once  the  improvement  plan  is  implemented,  the  SFA 
receives  a  closeout  letter  signaling  that  the  review  is  complete. 

New  Jersey  does  not  currently  have  any  method  for  aggregating  the 
nutrient  analyses.  State  officials  stated  that  they  preferred  to  wait 
and  see  what  information  FNS  requires  them  to  submit  before  devel¬ 
oping  a  system  to  use  for  aggregation.  They  would  like  FNS  to  pro¬ 
vide  a  protocol  for  a  database  before  they  create  anything.  All  hard 
copies  of  analyses  are  kept  at  the  state  office.  Currently,  no  elec¬ 
tronic  information  is  kept  on  reviews,  so  New  Jersey  would  also  need 
to  devise  a  system  to  send  the  data  to  FNS  electronically.  State  staff 
are  willing  to  do  what  FNS  needs  but  are  waiting  for  guidance  from 
FNS. 

New  Jersey  has  716  SFAs  and  had  completed  reviews  on  100  (14  per¬ 
cent)  as  of  August  1999.  About  90  percent  of  SFAs  in  New  Jersey  are 
using  enhanced  food-based  menu  planning  systems.  The  other  10 
percent  are  using  either  traditional  food-based  or  nutrient-based 
systems.  Although  no  longer  required  by  federal  regulations,  the 
state  of  New  Jersey  requires  nutrient  analyses  to  be  weighted  by  type 
of  entree  served. 

New  York 

In  New  York,  individual  reviewers  who  work  for  the  New  York  De¬ 
partment  of  Education  collect  the  necessary  information  from  the 
SFAs  during  the  site  visits  using  a  statewide  protocol.  The  level  of 
technical  assistance  varies,  however,  based  on  the  review  and  the 
SFA’s  need.  If  an  SFA  is  using  a  food-based  system,  the  reviewer 
performs  the  nutrient  analysis.  If  the  SFA  is  using  a  nutrient-based 
system,  the  SFA  is  responsible  for  its  own  nutrient  analysis  and  the 
reviewer  will  assess  that  analysis  as  well  as  the  SFA’s  competence  in 
doing  the  analysis.  After  the  analysis  is  reviewed,  a  correction  plan  is 
developed  with  the  SFA  and  a  hard  copy  of  the  final  review  is  filed  at 
the  State  Agency. 
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New  York  is  in  the  process  of  creating  a  Microsoft  Access  database  to 
aggregate  the  nutrient  analysis  information.  However,  making  the 
database  operational  is  not  a  priority.  State  officials  said  that  they 
prefer  to  wait  until  FNS  issues  its  requirements.  In  the  meantime, 
the  analyses  are  kept  in  hard  copy  at  the  state  office  and  a  summary 
of  each  analysis  is  kept  in  electronic  format.  However,  this  summary 
does  not  include  all  the  information  that  FNS  will  likely  require  on 
SMI  reviews.  Since  New  York  has  already  started  creating  a  database, 
it  should  not  be  a  difficult  task  to  modify  it  to  contain  all  the  neces¬ 
sary  data  elements  and  then  export  the  data  for  FNS. 

New  York  has  1,275  SFAs  and  had  completed  between  93  (7  percent) 
and  113  (9  percent)  SMI  reviews  as  of  August  1999.  About  50  percent 
of  SFAs  in  New  York  are  using  nutrient-based  analyses,  in  part  be¬ 
cause  the  state  staff  “put  a  lot  of  effort”  into  training  school  person¬ 
nel  on  how  to  perform  nutrient  analyses  correctly.  The  other  50  per¬ 
cent  are  fairly  evenly  split  between  enhanced  and  traditional  food- 
based  systems.  1  New  York  does  require  analyses  to  be  weighted. 

Texas 

Texas  has  a  different  system  for  completing  SMI  reviews  than  the 
other  six  states.  Unlike  the  other  state  agencies,  the  Texas  Education 
Agency  (TEA)  is  not  heavily  involved  in  SMI  reviews.  Instead,  TEA 
has  divided  Texas  among  20  Educational  Service  Centers  (ESCs), 
which  are  responsible  for  SMI  reviews  as  well  as  technical  assistance 
for  disabled  students,  multicultural  studies,  and  the  Public  Educa¬ 
tion  Information  Management  Systems,  which  collects  student  de¬ 
mographic  and  academic  performance,  personnel,  financial,  and  or¬ 
ganizational  information.  Each  ESC  has  a  Child  Nutrition  Office  that 
is  responsible  for  the  SMI  reviews.  Other  personnel  within  the  ESC 
perform  the  other  services  mentioned.  The  ESCs  are  independent 
entities  that  perform  reviews  and  provide  technical  assistance  to 
SFAs.  Therefore,  it  is  difficult  to  construct  a  comprehensive  descrip- 


Uhese  were  the  numbers  that  we  were  given  at  our  site  visit  in  August  1999.  New  York 
has  since  revised  these  numbers;  128  reviews  (10  percent)  completed  as  of  August 
1999.  About  40  percent  of  their  SFAs  are  using  nutrient-based  analyses.  The  en¬ 
hanced  food-based  option  was  selected  by  37  percent  of  the  SFAs,  and  the  traditional 
plan  was  selected  by  23  percent  of  the  SFAs.  The  numbers  in  Table  3.1  also  reflect 
what  we  heard  in  August  1999  and  would  be  updated  by  this  information. 
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tion  of  how  the  SMI  review  process  works  in  Texas  because  each  ESC 
is  autonomous  and  has  its  own  method  for  performing  reviews.  TEA 
does  not  currently  receive  any  nutrient  analysis  information  from  the 
ESCs,  and  therefore  no  aggregate  information  exists  at  the  state  level. 

The  ESC  staff  do  not  work  for  the  state  but  are  employees  of  the  ESC, 
which  is  an  independent  entity.  We  spoke  to  the  ESC  staff  in  Region 
XIII,  which  includes  the  area  around  Austin.  In  that  region,  the  ESC 
staff  perform  all  site  visits  and  the  nutrient  analysis.  The  standard 
process  is  for  the  ESC  staff  to  send  an  initial  letter  to  the  SEA  listing 
the  information  it  will  need  for  the  review.  Then,  a  second  letter  is 
sent  informing  the  SEA  of  the  dates  for  the  review  week  and  the  date 
of  the  site  visit.  Even  in  cases  where  an  SEA  is  using  a  nutrient-based 
system,  ESC  staff  will  redo  the  nutrient  analysis  rather  than  rely  on 
the  sea’s  analysis,  thus  doing  nutrient  analyses  on  every  SEA  that 
they  visit. 

Because  most  districts  are  not  ready  to  complete  a  final  analysis  of 
the  nutrients  in  their  meals,  the  analyses  often  have  to  be  done  sev¬ 
eral  times  before  they  are  correct.  Therefore,  officials  in  Texas  re¬ 
ferred  to  the  analysis  as  a  "living  document."  During  the  1998-1999 
school  year,  almost  all  analyses  in  Region  XIII  were  revised  after  they 
were  initially  done.  ESC  staff  reported  that  this  was  because  it  was 
the  first  year  they  did  SMI  reviews  and  SFAs  were  still  getting  used  to 
the  process. 

A  hard  copy  of  the  analysis  for  each  SEA  is  kept  by  the  ESC.  The  SFAs 
themselves  are  also  required  to  keep  a  copy  of  the  analysis  and  the 
improvement  plan.  General  information  about  all  analyses  per¬ 
formed  is  sent  to  the  state  annually.  However,  no  SFA-level  informa¬ 
tion  is  sent  to  the  state.  Of  all  the  states  we  spoke  to,  Texas  will  prob¬ 
ably  have  the  most  difficult  time  supplying  FNS  with  nutrient  analy¬ 
sis  data.  Since  the  state  staff  would  want  to  see  the  data  before 
sending  it  on,  they  would  first  need  to  collect  all  the  analyses  from 
the  20  ESCs  at  TEA  and  then  input  the  data  for  FNS  or  have  the  ESCs 
input  the  data  and  forward  it  to  TEA.  In  either  case,  the  process 
would  require  several  steps  that  TEA  is  not  currently  taking. 

The  TEA  and  the  ESCs  in  Texas  are  responsible  only  for  public  SFAs. 
The  20  ESCs  must  do  SMI  reviews  on  1,100  public  SFAs.  Because  no 
information  on  individual  reviews  is  sent  to  the  state,  it  is  not  possi- 
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ble  to  say  how  many  reviews  have  been  done  to  date  without  talking 
to  each  ESC  individually  (which  we  did  not  do).  About  50  percent  of 
SFAs  in  Texas  are  using  traditional  food-based  menu  planning  sys¬ 
tems.  Of  the  others,  about  30  percent  use  enhanced  food-based  and 
about  20  percent  use  nutrient-based  systems.  Texas  does  not  require 
analyses  to  be  weighted. 

Wisconsin 

In  Wisconsin,  a  consultant  collects  all  the  menus,  recipes,  and  other 
supporting  documents  from  the  SFA  during  the  site  visit,  unless 
these  materials  were  sent  prior  to  the  visit.  More  than  half  the 
schools  send  the  packets  ahead  of  time.  As  in  Georgia  and  Nebraska, 
the  state  employees  are  called  consultants.  Either  the  consultant 
performing  the  site  visit  or  a  central  consultant  at  the  state  office 
then  completes  the  nutrient  analysis.  If  materials  are  sent  prior  to 
the  visit,  then  the  analysis  can  be  completed  in  time  to  discuss  it  at 
the  site  visit.  Occasionally,  analyses  are  corrected  if  the  school  is  us¬ 
ing  a  nutrient-based  system  and  had  a  problem  doing  the  analysis  it¬ 
self  or  if  the  school  is  using  a  food-based  system  and  submitted  erro¬ 
neous  information  for  analysis.  When  an  analysis  is  corrected  under 
these  circumstances  the  updated  and  most  accurate  analysis  is 
recorded  in  the  state  spreadsheet.  Once  the  analysis  is  finished,  the 
consultant  works  with  the  SFA  to  form  a  correction  plan.  Once  the 
consultant  has  determined  that  the  district  has  begun  the  correction 
plan,  a  closure  letter  is  sent  to  the  SFA. 

Wisconsin  is  currently  using  a  Microsoft  Excel  spreadsheet  to  enter 
the  nutrient  analysis  information  for  monitoring  and  analysis  at  the 
state  level.  Hard  copies  of  the  analyses  are  also  kept  at  the  state  of¬ 
fice.  Assuming  FNS  would  accept  data  in  various  formats,  it  should 
be  a  simple  matter  for  Wisconsin  to  take  the  data  in  the  spreadsheet 
and  modify  it  for  FNS. 

Wisconsin  has  900  SFAs  and  had  completed  reviews  on  312  (35  per¬ 
cent)  as  of  June  1999.  Nearly  70  percent  of  SFAs  in  Wisconsin  use 
enhanced  food-based  systems.  Almost  30  percent  use  traditional 
food-based  systems,  and  less  than  5  percent  use  nutrient-based  sys¬ 
tems.  Wisconsin  requires  that  analyses  be  weighted  by  all  menu 
items. 
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SUMMARY  OF  SMI  REVIEW  INFORMATION 

In  this  section  we  present  the  information  on  all  seven  states  to¬ 
gether,  Table  3.1  lists  the  status  of  reviews  (June-August  1999)  in 
terms  of  the  SFAs  in  each  state  as  of  the  time  of  the  interview.  As  we 
mentioned  before,  the  agencies  in  Georgia  and  Texas  are  responsible 
only  for  public  SFAs;  therefore,  the  number  of  SFAs  listed  for  those 
states  represents  only  public  SFAs.  The  agencies  in  the  other  states 
are  responsible  for  both  public  and  private  SFAs. 

Table  3.1  suggests  that  the  states  are  at  varying  points  in  their  SMI 
review  efforts.  Georgia  and  Wisconsin  have  completed  more  than  a 
third  of  the  required  reviews.  New  Jersey  and  New  York  have  com¬ 
pleted  less  than  a  sixth. 

Interpretation  of  the  percent  of  reviews  completed  is  complicated  by 
the  nature  of  the  review  process.  Not  all  SMI  reviews  that  have  been 
performed  had  been  logged  into  state  systems  at  the  time  of  the  in¬ 
terviews.  Therefore,  some  states  had  to  estimate  the  number  of  re¬ 
views  that  were  done  in  the  1998-1999  school  year.  In  the  table  be¬ 
low,  the  numbers  represent  the  actual  completed  reviews.  Numbers 
in  parentheses  represent  the  estimated  number  and  percent  of  re¬ 
views  that  will  be  completed  once  all  reviews  for  that  school  year  are 
logged  in  and  exit  conferences  are  finished. 

Table  3.1 

Summary  of  Status  of  SMI  Reviews  in  the  Seven  States 


State 

Number 
of  SFAs 

Number  of  SMI  Reviews 
Completed  as  of  Sum¬ 
mer  1999 

Percentage  of  SMI  Re¬ 
views  Completed  as  of 
Summer  1999 

California 

1,496 

206  (307) 

14  (21) 

Georgia 

180 

78 

43 

Nebraska 

506 

103 

20 

New  Jersey 

716 

100 

14 

New  York 

1,275 

93(113) 

7(9) 

Texas 

1,100 

188 

17 

Wisconsin 

900 

312 

35 

NOTES:  Numbers  in  parentheses  indicate  the  estimated  number  or  percentage  of  re¬ 
views  completed  once  they  are  all  logged  in;  New  York  figures  are  the  revised  numbers 
sent  after  our  visit. 
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Table  3.2  summarizes  the  information  collected  on  how  the  SMI  re¬ 
views  were  conducted  within  the  seven  states.  In  some  states  the 
reviewer  does  the  nutrient  analysis.  In  other  states,  the  nutrient 
analysis  is  done  by  someone  else — a  member  of  the  central  office  or 
an  outside  contractor.  Some  states  use  a  mixture  of  the  two  ap¬ 
proaches.  The  “Weighted  Analysis"  column  indicates  whether  a  state 
is  using  that  method. 

Most  of  the  states  received  the  data  from  each  nutrient  analysis  in 
the  State  Agency  office.  The  column  entitled  “State  Has  Data"  in 
Table  3.2  shows  whether  the  state  office  receives  the  data  and 
whether  they  have  compiled  it  into  any  type  of  central  database. 

All  State  Agency  personnel  or  State  Agency  contractors  in  these  seven 
states  use  NutriKids  software.  However,  SFAs  and  others,  such  as 
food  service  management  companies  that  perform  their  own  com¬ 
puter  analyses,  may  use  different  software.  Therefore,  not  all  nutri¬ 
ent  analyses  in  each  state  are  necessarily  done  on  NutriKids. 

Table  3.3  shows  the  reviewers’  employer  and  location.  In  every  state 
except  Texas,  the  reviewers  work  directly  for  the  State  Agency,  al¬ 
though  they  are  not  always  based  at  the  central  office. 

In  this  table,  “State — Centralized"  refers  to  reviewers  who  work  out 
of  the  main  office  of  the  State  Agency.  “State — Distributed"  refers  to 
reviewers  who  work  for  the  State  Agency  but  are  located  in  other 

Table  3.2 

Summary  of  How  SMI  Reviews  Were  Performed  in  the  Seven  States 


State 

Who  Does  Analysis 

Reviewer  Other 

Weighted 

Analysis? 

State  Has 

Data?* 

Software  Used 

California 

X 

X 

Some 

Y,DB 

NutriKids 

Georgia 

X 

X 

Y 

Y,NDB 

NutriKids 

Nebraska 

X 

Y 

Y,DB 

NutriKids 

New  Jersey 

X 

Y 

Y.NDB 

NutriBCids 

New  York 

X 

X 

Y 

Y,DB 

NutriKids 

Texas 

X 

N 

N 

NutriKids 

Wisconsin 

X 

X 

Y 

Y,DB 

NutriKids 

*DB  =  database  or  spreadsheet;  NDB  =  no  database  or  spreadsheet 
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parts  of  the  state  away  from  the  central  office.  "Contract — Dis¬ 
tributed"  refers  to  reviewers  who  do  not  work  for  the  State  Agency 
and  are  spread  out  around  the  state. 

Table  3.4  shows  the  summary  of  menu  planning  systems  that  dis¬ 
tricts  use  in  each  of  the  seven  states.  In  five  of  the  seven  states,  more 
than  80  percent  of  SFAs  are  using  food-based  systems,  either 
"Traditional”  or  "Enhanced."  In  three  of  the  seven  states,  more  than 
15  percent  of  SFAs  are  using  "Nutrient  Analysis."  In  New  York,  nearly 
half  of  the  SFAs  are  using  "Nutrient  Analysis.”  In  California  and 
Texas,  nearly  one  in  five  SFAs  are  using  "Nutrient  Analysis." 

Table  3.3 

Summary  of  SMI  Reviewers’  Employer  and  Location  in  the  Seven  States 


State 

State — Centralized 

State — Distributed 

Contract — Distributed 

California 

X 

X 

Georgia 

X 

X 

Nebraska 

X 

New  Jersey 

X 

New  York 

X 

X 

X 

Texas 

X 

Wisconsin 

X 

Table  3.4 

Summary  of  Menu  Planning  Systems  Used  for  SMI  Reviews  in  the  Seven 

States 


State 

Menu  Planning  System  Used 
(Rough  Estimates  of  Percent  of  Districts) 

Nutrient 

Analysis 

Enhanced 

Food-Based 

Traditional 

Food-Based 

Other 

California 

18 

37 

20 

25 

Georgia 

? 

? 

95 

0 

Nebraska 

4 

80 

16 

0 

New  Jersey 

3 

91 

6 

0 

New  York 

48 

28 

23 

0 

Texas 

17 

30 

53 

0 

Wisconsin 

3 

68 

28 

0 

NOTE:  ?  =  State  was  unable  to  provide  these  numbers. 


Chapter  Four 

HOW  DATA  ELEMENTS  COULD  BE  COLLECTED 


Having  considered  what  should  be  reported  and  current  state  review 
procedures,  in  this  chapter  we  turn  to  the  core  of  this  report:  How 
states  could  most  easily  report  the  results  of  their  review  efforts  to 
USDA.  We  begin  by  asking  two  questions:  Who  should  do  the  report¬ 
ing?  When  in  the  review  process  should  the  report  be  sent  to  USDA? 
We  then  consider  the  technical  details  of  how  the  reporting  should 
be  done.  On  the  state  side,  we  consider  file  formats  and  methods  for 
forwarding  the  data  to  USDA.  On  the  USDA  side,  we  consider  how  to 
store  and  analyze  the  data.  Throughout,  we  give  careful  considera¬ 
tion  to  minimizing  the  burden  on  the  states. 

WHO  SHOULD  REPORT  WHEN? 

Who  should  report  the  state's  review  information  and  when  in  the 
review  process  should  that  person  do  so?  To  consider  the  broadest 
possible  range  of  approaches  to  state  reporting,  we  want  to  begin 
with  the  point  at  which  there  exists  a  nutrient  analysis  that  could  be 
provided  to  USDA.  To  do  that,  we  need  to  consider  when  a  review  is 
complete. 

As  discussed  in  the  previous  chapter,  state  procedures  vary.  For  the 
purposes  of  this  discussion,  it  is  enough  to  note  that  the  materials 
from  the  review  are  collected  by  someone  and  entered  into  the  nutri¬ 
ent  analysis  software  (sometimes  the  person  making  the  on-site  visit, 
sometimes  a  state  employee  in  a  central  office,  sometimes  a  private 
nutritionist).  The  results  of  the  nutrient  analysis  are  available  essen¬ 
tially  immediately  after  all  of  the  information  (i.e.,  menus  and  nutri¬ 
tional  content  of  ingredients)  is  entered — a  few  seconds  to  do  the 
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analysis,  a  few  more  seconds  to  save  or  print  the  results.  In  practice, 
it  often  takes  several  cycles  of  data  collection  (by  field  staff),  data  en¬ 
try,  and  review  (by  a  senior  staff  person)  to  collect  all  of  the  necessary 
information  (menus,  specific  products  used,  nutritional  content  of 
products)  and  to  clarify  apparently  erroneous  information.  For  ex¬ 
ample,  the  data  entry  person  may  begin  input  and  realize  that  a 
menu  is  missing  or  a  food  item's  nutrient  content  is  not  included.  In 
these  cases,  the  state  staff  contacts  the  SFA  again  for  this  information 
before  data  input  can  be  completed.  For  the  discussion  here,  that  is 
an  unimportant  detail. 

Similarly,  under  some  circumstances,  some  states  revise  their  re¬ 
views.  For  example,  in  Georgia,  about  half  of  the  nutrient  analyses 
are  redone  after  an  improvement  plan  is  implemented.  In  most 
other  states,  analyses  are  revised  only  if  an  error  is  found,  either  with 
data  entry  or  with  the  materials  that  the  SFAs  provided. 

To  monitor  the  content  of  meals  actually  being  served,  USDA  would 
want  the  results  of  the  final  reviews.  Most  states  that  enter  the  nutri¬ 
ent  analysis  data  into  a  database  or  spreadsheet  use  the  initial  re¬ 
view.  Thus,  it  should  be  possible  to  report  the  results  of  the  review 
immediately  after  completion  of  the  nutrient  analysis.  It  should  not 
be  necessary  to  wait  for  the  completion  of  any  revised  review,  nor  for 
the  development  of  the  improvement  plan,  nor  for  the  official  closing 
of  the  review.  Since  only  the  initial  analysis  is  considered  part  of  an 
SMI  review  according  to  FNS,  states  should  only  report  information 
from  this  analysis  to  FNS.  Only  in  cases  where  an  error  was  found 
with  inputting  the  information  in  the  analysis  software  would  a 
"revised”  analysis  be  appropriate.  In  states  such  as  Georgia  where 
analyses  are  revised  based  on  the  implementation  of  improvement 
plans,  USDA  would  want  the  results  of  the  pre-revision  nutrient 
analysis.  Therefore,  FNS  should  make  clear  in  directions  to  the  states 
that  the  original  analysis  is  the  appropriate  one  to  report,  even  if  this 
is  not  when  the  state  considers  the  review  completed. 

To  allow  for  the  broadest  range  of  options,  we  should  consider  hav¬ 
ing  any  of  the  people  who  have  access  to  the  nutrient  analysis  after 
its  completion  forward  the  review  to  USDA.  Depending  on  the  state, 
the  list  of  such  people  might  include  the  person  who  inputs  the  data 
and  performs  the  nutrient  analysis  (perhaps  the  field  staff  person, 
perhaps  a  state  employee  in  a  central  office,  perhaps  a  contractor), 
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the  person  who  reviews  the  computer  output,  or  the  clerk  who  files 
the  analysis  (though  the  clerk  will  often  not  get  the  file  until  the  re¬ 
view  is  complete,  sometimes  well  after  the  nutrient  analysis  is  com¬ 
plete). 

In  practice,  the  consideration  of  such  options  was  uniformly  and  ve¬ 
hemently  rejected  by  every  state  we  visited.  Senior  state  officials  all 
said  that  they  were  responsible  for  the  SMI  reviews.  If  the  results  of 
the  nutrient  analysis  were  to  be  reported  to  USDA,  they  would  want 
to  review  the  analysis  or  have  a  designated  person  in  their  office  do 
so  before  it  was  forwarded  to  USDA. 

This  state  reaction  has  important  and  unfortunate  consequences  for 
USDA.  USDA’s  stated  goal  is  to  have  states  report  data  with  minimal 
additional  effort  or  disruption  of  their  current  routine.  At  least  in 
Texas,  these  two  goals  appear  to  be  mutually  exclusive.  The  Texas 
Education  Agency  does  not  currently  collect  the  results  of  reviews. 
Instead,  the  individual  reviews  are  held  by  the  Educational  Service 
Centers  (ESCs).  Nevertheless,  even  Texas  officials  stated  that  they 
would  not  be  willing  to  forward  the  results  of  nutrient  analyses  to 
USDA  without  revie^ving  them  centrally.  Thus,  any  USDA  reporting 
requirement  will  force  Texas  to  collect  and  review  the  nutrient  analy¬ 
ses  centrally. 

While  in  principle  we  could  consider  several  alternatives  as  to  when 
in  the  process  the  report  should  be  sent  to  USDA  and  by  whom,  in 
practice,  it  seems  clear  that  reporting  will  be  done  by  central  office 
staff.  Furthermore,  this  line  of  reasoning  strongly  suggests  that  the 
reporting  will  be  done,  not  when  the  nutrient  analysis  is  completed, 
but  when  the  review  (including  the  completion  of  the  improvement 
plan)  is  completed.  Furthermore,  it  will  sometimes  be  true  that  by 
the  time  the  review  is  completed,  there  will  be  not  only  the  initial 
nutrient  analysis  but  also  a  revised  nutrient  analysis.  USDA  will  need 
to  give  clear  instructions  as  to  which  review  it  wants.  We  recom¬ 
mend  that  USDA  explicitly  request  the  initial  review. 

FILE  FORMATS  AND  STATE  DATABASES 

USDA’s  projected  uses  of  the  data  require  that  it  receive  not  aggre¬ 
gated  data  (as  is  currently  true  for  Coordinated  Review  Efforts 
(CREs),  but  instead  information  on  each  review.  Happily,  this  need 
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appears  to  be  consistent  with  state  desires.  States  have  complained 
that  they  have  little  use  for  the  aggregate  CRE  data  and  that  aggregat¬ 
ing  the  results  is  therefore  an  unnecessary  burden. 

Having  tentatively  concluded  that  reviews  will  be  forwarded  to  USDA 
by  some  member  of  the  state's  central  office  staff,  we  turn  to  the  file 
format  in  which  the  data  should  be  forwarded  to  USDA  and  the 
closely  related  issue  of  state  databases.  There  appear  to  be  two 
broad  approaches  to  these  issues: 

•  States  could  create  a  database  for  their  own  use.  At  some  interval 
(presumably  annually),  the  states  could  forward  the  content  of 
the  database  to  USDA. 

•  Alternatively,  states  could  forward  the  information  to  USDA  re¬ 
view  by  review.  In  this  case,  states  might  find  it  useful  to  access 
USDA's  database. 

We  consider  each  of  these  approaches  and  mixtures  of  the  two. 

State  Databases 

Some  states  (California,  Nebraska,  New  York,  and  Wisconsin)  are  al¬ 
ready  working  on  their  own  local  databases  or  spreadsheets.  They 
record  the  results  of  each  review  into  the  database  as  it  is  completed. 
They  then  use  the  database  locally  to  monitor  the  status  of  reviews 
and  the  results  of  the  reviews.  We  learned  of  databases  being  con¬ 
structed  in  Microsoft  Excel  (spreadsheet),  Microsoft  Access  (low-end 
database),  and  Oracle  (high-end  database).^ 

Several  other  states  indicated  that  they  expected  to  move  to  a  local 
database  in  the  intermediate  term.  One  state  (New  York)  already  had 
a  local  database  project  underway.  Another  state  (New  Jersey)  was 
deferring  its  local  database  project  until  it  received  clearer  guidance 
from  USDA  on  what  should  be  collected.  Some  states  (New  York, 
New  Jersey,  Georgia)  indicated  that  they  would  welcome  (and  per- 


^Wisconsin  uses  an  Excel  spreadsheet;  California,  Nebraska,  and  New  York  are  in 
various  phases  of  developing  an  Access  database;  New  York  uses  Oracle  for  some  of  its 
database  needs,  though  it  has  not  developed  a  full  database  for  SMI. 
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haps  expect)  USDA-provided  software  for  the  construction  of  a  local 
database.^ 

The  existence  of  a  local  database  is  useful  for  USDA  because  the  data 
already  exists  in  computerized  form  and  it  should  be  relatively  easy 
for  the  state  to  forward  the  review  information  to  USDA.  The  state 
would  simply  forward  the  computer  file  to  USDA  (perhaps  with  some 
documentation  as  to  the  meaning  of  fields).  Some  simple  pro¬ 
gramming  might  be  needed  to  read  the  state  file  into  USDA’s  na¬ 
tional  database,  but  doing  so  would  be  relatively  straightforward. 
Standard  (and  inexpensive)  software  products  (e.g.,  DBMS/Copy, 
Stat/Transfer^)  will  perform  the  file  format  translations  (e.g.,  from 
Excel  or  Access  into  Oracle  or  SAS).  Some  software  packages  will  do 
the  file  format  translations  internally  (e.g.,  SAS  will  read  Excel  and 
SQL  databases  such  as  Oracle). 

The  file  format  translations,  however,  are  only  the  start  of  the  task. 
Once  the  files  are  in  the  final  file  format,  there  is  then  the  issue  of  re¬ 
naming  variables  and  recoding  values  so  that  the  state  data  can  be 
merged  into  the  USDA  database  to  yield  analyzable  data.  For  this 
task,  the  effort  required  by  USDA  to  read  the  data  into  its  database 
will  be  approximately  proportional  to  the  number  of  different 
database  systems  the  states  use.  As  long  as  the  file  formats  do  not 
change  much  from  year  to  year,  50  states  is  a  feasible  number  of  for¬ 
mats  to  process.  A  programmer  of  moderate  skill  could  probably  do 
most  formats  in  an  hour  or  two. 

Clearly,  however,  it  would  be  much  simpler  for  USDA  to  provide  the 
states  with  a  program  that  they  may  choose  to  use.  Several  states 
said  they  preferred  that  USDA  provide  the  program,  and  if  it  did  so 
they  would  use  it.  Given  the  goal  to  minimize  the  work  for  both  the 
states  and  USDA,  supplying  a  sample  program  to  the  states  is  attrac¬ 
tive. 

We  have  created  a  sample  local  database  in  Microsoft  Access  with 
data  input  screens  and  rudimentary  data  input  checks.  To  make  it 


sample  local  database  is  available  at  http://www.rand.org/organization/drd/ 
labor/foodnupolicy.html. 

^See  http://www.conceptual.com/dbmscopy.htm  or  http://www,stattransfer.com/ 
for  more  information. 
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fully  functional,  it  probably  requires  some  simple  documentation 
(under  ten  pages),  some  more  data  input  checks,  some  simple  re¬ 
porting  functions,  and  a  simple  file  export  function  (see  below).  The 
time  required  to  add  these  functions  is  probably  less  than  a  month. 

Some  states  will  not  use  a  USDA  database  because  they  have  already 
developed  a  database  with  which  they  are  happy.  For  some  states, 
this  database  is  integrated  with  other  parts  of  their  system  (e.g.,  it 
also  holds  financial  data  or  CRE  data). 

For  those  states,  USDA  will  need  to  make  a  policy  choice.  Someone 
will  need  to  do  the  file  translations.  On  whom  does  USDA  want  to 
put  the  cost  of  making  the  data  compatible?  USDA  could  impose 
those  costs  on  the  states.  In  that  case,  USDA  could  publish  specifi¬ 
cations  (for  the  names  of  variables  and  the  coding  scheme)  and  then 
require  the  states  to  provide  data  according  to  the  specification.  Pre¬ 
sumably,  the  USDA-provided  program  would  automatically  satisfy 
the  specifications,  which  would  be  another  incentive  for  the  states  to 
use  the  USDA  program.  This  approach  seems  inconsistent  with  the 
stated  USDA  desire  to  minimize  the  cost  to  the  states. 

Alternatively,  USDA  could  simply  require  the  states  to  deliver  some 
computer  file  (see  below  for  details).  USDA  would  then  have  to 
translate  the  files  itself.  If  USDA  provides  a  program  that  the  states 
can  use  at  their  option,  it  seems  likely  that  many  (perhaps  most) 
states  will  do  so,  thus  the  cost  to  USDA  will  not  be  too  large.  Given 
USDA’s  stated  desire  to  minimize  the  costs  to  the  states,  it  seems 
preferable  for  USDA  to  offer  to  take  the  data  in  any  well-defined  and 
documented  file  format. 

No  State  Database 

Not  all  states  currently  have  a  database,  and  it  is  not  clear  that  it  is 
necessary  for  each  state  to  maintain  a  database.  In  the  Internet  era, 
USDA  could  maintain  the  database  and  states  could  access  it  as 
needed  to  generate  reports. 

Under  this  model,  states  would  report  the  results  of  individual  nutri¬ 
ent  analyses  to  USDA  as  each  review  is  completed.  A  natural  way  to 
do  so  would  be  over  the  Internet.  USDA  would  provide  a  web  site 
with  a  data  input  form,  and  the  clerk  in  the  state  central  office  would 
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access  the  web  site  and  input  the  data.  We  provide  a  sample  Mi¬ 
crosoft  Front  Page  program  for  the  data  input  at  the  RAND  web  site,^ 
and  we  discuss  data  input  and  transmission  in  the  next  section. 

Ideally,  the  nutrient  analysis  software  (e.g.,  NutriKids)  would  gener¬ 
ate  a  file  with  all  the  required  information.  As  we  noted  earlier,  cur¬ 
rent  software  provides  some,  but  not  all,  of  the  required  information. 
In  that  case,  the  clerk  could  simply  forward  the  output  from  the 
computer  program. 

TRANSMITTING  THE  DATA 

Even  with  agreement  as  to  file  formats,  there  remains  the  question  of 
how  the  data  should  actually  get  to  USDA.  Many  options  are  possi¬ 
ble.  At  the  simplest  level,  the  states  could  send  a  computer  disk  with 
the  data  to  USDA  through  the  mail.  The  files  will  be  quite  small,  so  a 
3.5-inch  computer  disk  is  feasible  and  inexpensive,  but  disks  are  no¬ 
torious  for  being  corrupted  easily.  There  is  a  danger  of  data  being 
sent  in  a  timely  manner  but  FNS  not  being  able  to  use  them  because 
of  corrupt  disks. 

Paper  reports  are  in  principle  possible  but  seem  unnecessarily  cum¬ 
bersome.  Using  paper  reports  would  require  USDA  to  rekey  the  in¬ 
formation,  which  is  inefficient. 


^This  program  is  not  a  complete  prototype  but  demonstrates  the  capability  for  remote 
data  entry.  The  current  program,  however,  does  not  allow  editing  of  already  inputted 
data.  That  step  will  be  required  of  any  production  system.  Errors  in  data  entry  are  not 
uncommon  and  they  would  need  to  be  corrected  (presumably  by  the  state). 

Some  subtle  data  safeguarding  issues  are  raised  by  doing  such  data  editing  over  the  In¬ 
ternet.  Any  Internet  system  would  require  some  form  of  security  (e.g.,  password  pro¬ 
tections  to  prevent  unauthorized  access  to  or  modification  of  the  data).  Only  USDA 
and  the  reporting  state  should  be  able  to  edit  information  for  a  state — not  other  states 
and  not  outsiders. 

Similarly,  USDA  would  need  to  develop  policies  about  who  could  analyze  which  parts 
of  the  data.  Clearly,  USDA  would  want  to  be  able  to  analyze  the  data.  If  USDA  is 
maintaining  a  database  of  state  data,  it  seems  reasonable  that  the  states  should  be  able 
to  access,  analyze,  and  download  their  own  data.  USDA  would  need  to  decide  if  the 
filings  were  to  be  considered  public  information  such  that  other  states  or  nongovern¬ 
mental  organizations  would  be  allowed  to  access  and  analyze  (but  not  modify)  the 
data.  The  federal  Freedom  of  Information  Act  may  apply. 
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All  states  reported  that  they  had  (or  would  soon  have)  e-mail  and 
Internet  access  for  their  field  staff  and  for  their  central  office.  E-mail 
technology  for  the  transfer  of  computer  files  is  today  quite  robust.  If 
states  are  going  to  be  required  to  provide  USDA  with  files,  e-mail  is 
likely  to  be  the  lowest  cost  and  easiest  option  for  both  USDA  and  the 
states. 

On  the  USDA  side,  the  standard  approach  to  e-mail  submission  is  the 
creation  of  a  dummy  e-mail  account  (perhaps  SMI-input@usda.gov). 
States  would  send  their  submissions  to  this  account.  Whomever  at 
USDA  is  currently  responsible  for  the  data  would  be  given  authority 
to  access  the  e-mail  messages.  Such  a  dedicated  account  would  get 
the  state  submissions  out  of  the  stream  of  other  e-mail  coming  to  the 
designated  USDA  staffer.  This  approach  also  allows  USDA  to  change 
the  dedicated  staffer  without  informing  or  inconveniencing  the 
states. 

Similarly,  if  the  states  provide  data  review- by- review,  input  based  on 
Internet  forms  seems  most  attractive.  All  states  we  talked  to  have  In¬ 
ternet  access,  and  the  software  could  be  made  easy  to  use.  Input 
should  be  completable  in  well  under  fifteen  minutes  (perhaps  five 
minutes).  States  would  submit  their  data  using  an  Internet  form  that 
looks  similar  to  the  Access  screen  shot  in  Chapter  2.  RAND’s  Survey 
Research  Group  has  used  such  Internet  data  input  for  some  surveys 
of  professionals.  Many  high-tech  marketing  firms  conduct  market 
surveys  in  this  way.^  All  e-commerce  sites  have  a  similar  data  input 
operation  (e.g.,  Amazon.com,  eBay.com,  and  eToys.com). 

One  option  for  setting  up  an  Internet  form  is  to  create  the  web  site 
(an  HTML  document)  with  Microsoft  Access  in  the  background.  This 
option  allows  USDA  to  send  the  data  directly  into  a  Microsoft  Access 
database  as  they  are  entered.  However,  this  has  some  limitations 
since  Access  is  not  really  capable  of  handling  multiple  users  at  the 
same  time.  Another  option  is  to  use  an  HTML  document  with  an 
application  that  can  handle  multiple  users,  such  as  Oracle  Personal 
Server,  Microsoft  SQL  Server,  or  Visual  InterDev.  USDA  would  just 
need  to  have  the  appropriate  program  loaded  onto  a  server.  In  all  of 


^See  “The  Survey  Says. . in  PC  Magazine,  May  4,  1998,  for  more  information  on  web- 
based  surveys  and  using  the  data  in  Microsoft  Access. 
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these  cases,  the  web  form  would  look  just  like  a  regular  web  page  to 
the  person  doing  the  data  entry. 

The  only  problem  that  a  web-based  form  might  pose  is  the  currency 
of  the  states’  web  browsers.  If  FNS  can  be  sure  that  all  the  states  are 
using  something  reasonably  current,  then  web  site  development  will 
be  relatively  straightforward.  If  many  states  are  using  outdated 
browsers,  development  might  be  more  complicated.  Depending  on 
how  competent  the  developer  is  (e.g.,  how  well  that  person  under¬ 
stands  the  limitations  of  older  or  obscure  web  browsers),  this  could 
have  an  impact  on  the  users.  However,  this  problem  is  relatively 
small  since  many  updated  web  browsers  are  available  over  the  Inter¬ 
net  for  free.  FNS  would  merely  need  to  instruct  the  states  to  obtain  a 
current  web  browser. 

Such  Internet  input  guarantees  consistent  data  and  minimizes  the 
cost  to  USDA  in  terms  of  tracking  files  submitted  and  reformatting 
information.  Furthermore,  the  system  could  easily  be  constructed 
such  that  a  state  could  download  data  to  a  local  (i.e.,  in  the  home 
state)  computer  for  further  analysis  (e.g.,  by  Microsoft  Excel  or  Mi¬ 
crosoft  Access). 

PUTTING  IT  ALL  TOGETHER 

USDA’s  stated  goal  is  to  collect  the  data  at  minimal  cost  to  the  states. 
Therefore,  it  seems  appropriate  for  USDA  to  accept  the  data  in  any 
reasonable  format  that  the  states  are  willing  to  provide.  Therefore, 
we  recommend  that  USDA  accept  data  in  all  of  the  following  forms: 

1.  USDA  File  Formats:  USDA  develops  and  offers  to  the  states  a 
simple  program  allowing  the  creation  and  analysis  of  a  local  state 
database.  The  RAND-provided  Microsoft  Access  program  is  a 
start  toward  such  a  program.  States  then  forward  the  results  an¬ 
nually,  using  the  CRE  deadlines,  to  a  dedicated  e-mail  account  at 
USDA.6 


®To  encourage  states  to  use  USDA's  program  (and  thus  to  minimize  USDA’s  pro¬ 
gramming  costs),  USDA  might  offer  to  pay  for  programming  time  to  read  existing  state 
databases  (i.e.,  the  information  in  the  database)  into  the  USDA  program. 
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2.  Other  File  Formats:  USDA  accepts  from  the  states  any  computer 
file  they  wish  to  submit.  The  only  requirements  are  that  the  file 
contain  all  of  the  required  data  elements  and  that  the  state  docu¬ 
ment  the  data  sufficiently  well  that  USDA  can  read  the  data  into 
its  database.  USDA  would  pay  a  programmer  to  read  the  state 
files  into  its  database.  Given  that  most  states  would  probably  not 
choose  this  option,  the  required  time  would  probably  be  only  a 
week  or  two.^  Again,  the  file  would  be  delivered  annually  by  e- 
mail  to  a  dedicated  e-mail  account.® 

3.  Internet  Input:  USDA  develops  an  Internet-based  data  input  pro¬ 
gram  through  which  the  states  could  provide  the  results  of  nutri¬ 
ent  analyses  on  a  review-by-review  basis.  The  Internet  input  op¬ 
tion  is  the  least  clear:  Developing  the  software  will  be  more  ex¬ 
pensive  than  the  other  options,  and  it  is  not  clear  whether  the 
states  would  use  the  option. 


CUSTOMIZING  DATABASES  FOR  THE  STATES 

One  issue  that  must  be  considered  when  deciding  the  reporting  sys¬ 
tem  is  how  the  states  will  want  to  use  these  data  for  their  own  pur¬ 
poses.  Some  states  already  have  a  system  in  place  for  reviewing  their 
SMI  data.  Others  do  not  have  anything  in  place  and  do  not  plan  to 
develop  a  system  until  they  are  required  to  do  so.  However,  if  states 
are  required  to  report  data  to  FNS,  they  will  undoubtedly  want  to 
have  use  of  that  data  within  the  state  as  well.  Therefore,  this  raises 
the  issue  of  ensuring  that  the  states  are  able  to  customize  the 
database  for  their  own  use. 

If  states  already  have  their  own  system  and  they  will  be  sending  their 
data  to  FNS  in  their  current  application  (i.e.,  Microsoft  Access  or  Ex- 


^In  the  first  year,  the  cost  of  these  two  options — USDA  program  development  for  the 
states  to  use  or  state  data  files  on  receipt — would  be  similar.  Thereafter,  the  first  op¬ 
tion  is  likely  to  be  cheaper.  If  states  convert  to  the  USDA  program,  then  it  is  easy  for 
USDA  to  change  the  submission  rules  (e.g.,  to  add  new  fields,  to  change  the  codes,  to 
improve  the  data  validation).  However,  if  the  states  customize  USDA’s  program, 
USDA  revisions  to  its  program  may  require  additional  work  by  the  states  to  redo  their 
customizations.  Because  of  this  and  to  minimize  the  need  for  customization,  USDA 
should  poll  the  states  about  additional  program  features  desired. 

®So  that  USDA  would  not  have  to  do  conversions  more  than  once  a  year,  the  informa¬ 
tion  should  be  sent  annually. 
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cel,  Oracle),  then  they  can  easily  customize  the  system  to  include 
other  data  elements  for  their  internal  analyses.  They  would  then  just 
need  to  send  the  required  data  elements  to  FNS. 

If  states  were  to  use  a  protocol  given  to  them  by  FNS  (e.g.,  the  Access 
database  RAND  has  designed),  states  could  customize  the  system  in 
much  the  same  way  they  would  their  own  system.  Again,  they  would 
just  need  to  send  the  appropriate  data  elements  to  FNS. 

In  the  case  of  a  web -based  system,  two  issues  pose  difficulty  for  cus¬ 
tomization.  The  first  concerns  getting  their  own  data  without  others 
also  having  access  to  it.  Allowing  states  to  query  any  data  in  a  web- 
based  system  is  not  difficult,  but  allowing  states  to  access  only  their 
own  data  is  probably  more  desirable  but  requires  some  program¬ 
ming.  The  second  issue  is  actual  customization.  If  states  were  able  to 
download  a  plain  text  file  from  a  web-based  system,  they  could  po¬ 
tentially  merge  it  with  the  other  data  elements  that  they  want  to  add. 
However,  this  requires  an  identifier  for  the  review  to  have  something 
on  which  to  merge.  It  is  clear  that  the  web -based  system  would  be 
the  most  difficult  for  states  to  customize.  However,  some  states  that 
do  not  currently  have  systems  in  place  may  wish  to  use  this  since 
they  may  not  see  a  need  for  additional  analyses  using  other  data  el¬ 
ements.  A  web-based  system  would  save  them  having  to  develop  or 
maintain  anything  new. 

MANAGING  THE  DATABASE  AT  USDA 

We  conclude  with  some  thoughts  about  the  database  at  USDA. 
There  are  about  20,000  SFAs  in  the  country.  With  a  five-year  review 
cycle  and  some  SFAs  using  multiple  menu  planning  systems,  USDA 
would  expect  on  the  order  of  4,000  new  records  per  year.  Each 
record  would  have  between  20  and  30  data  elements. 

This  is  a  moderate- sized  database  that  can  be  run  on  the  standard 
desktop  personal  computer  purchased  today  (late  1999).^  Clearly,  e- 
mail  and  web  access  would  be  needed.  For  software,  USDA  could 


®One  example  is  Dell’s  low-level  Dimension  L433CD  with  a  433  MHz  Celeron  proces¬ 
sor,  64MB  RAM,  and  4.3GB  hard  disk.  List  price  is  $1,299  as  of  November  1999. 
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use  Microsoft  Access,  an  Oracle  database,  or  a  more  conventional 
statistical  package  such  as  SAS. 

The  tabulations  required  by  USDA  are  straightforward  and  could  be 
precoded.  In  the  language  of  SAS,  they  are  “frequencies”  with 
“subsetting  if'  and  “by”  statements.  The  systems  should  probably 
include  the  possibility  of  ad  hoc  queries.  QBE  (Query  by  Example)  is 
the  easiest  way  to  specify  such  queries.  An  analyst  with  rudimentary 
experience  in  statistical  analysis  (e.g.,  those  required  to  complete  a 
masters  program  in  public  policy)  should  be  able  to  program  the  re¬ 
quired  queries  without  major  assistance. 


Chapter  Five 

DISCUSSION  OF  THE  CONTENT  OF  SPECIFIC  MEALS 


The  discussion  of  the  previous  four  chapters  has  considered  the 
technical  question  of  what  information  USDA  should  collect  about 
SFA-level  SMI  reviews  and  how  that  information  should  be  for¬ 
warded  to  USDA.  In  this  chapter,  we  briefly  consider  the  broader  is¬ 
sues  of  how  the  SMI  effort  in  general  (especially,  state  reporting  to 
USDA)  advances  the  statutory  goals  of  the  NSLP  (in  particular,  the 
compliance  of  NSLP  meals  with  the  DGA) . 

We  begin  with  a  discussion  of  two  positive  effects  of  the  current  re¬ 
view  procedures:  First,  the  experience  of  preparing  for  a  review  itself 
raises  consciousness  of  nutritional  issues;  and  second,  as  intended, 
the  results  of  the  review  lead  to  improvements  in  the  nutritional 
content  of  meals. 

We  then  consider  three  specific  concerns  about  the  quality  of  the 
data  that  are  collected.  Combined  with  the  required  effort,  these 
concerns  lead  some  state  officials  to  question  the  value  of  applying 
limited  staff  to  the  SMI  effort.  For  example,  New  York  state  staff 
commented  that  “too  much  of  [our]  resources  are  directed  toward 
artificial  data  gathering,  rather  than  feeding  kids.” 

In  particular,  three  considerations  suggest  that  the  present  SMI  re¬ 
view  program  may  not  be  ideal  for  achieving  the  goals  set  out  in  the 
USDA  Strategic  Plan — especially  that  of  ensuring  that  school  meals 
are  consistent  with  the  Dietary  Guidelines  for  Americans  and  the 
Recommended  Daily  Allowances.  The  first  issue  concerns  the  qual¬ 
ity  of  the  review  information  for  the  meals  reviewed.  The  second  is¬ 
sue  concerns  the  representativeness  of  the  meals  being  reviewed  at 
the  SFA  level  (i.e.,  how  useful  is  the  information  for  parents  in  a  given 
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SFA  about  the  meals  being  served  by  their  SFA).  The  third  issue  con¬ 
cerns  the  representativeness  of  meals  being  reviewed  at  the  state 
level  (i.e.,  how  useful  would  some  average  of  information  about  all 
meals  across  groups  of  schools  be  to  USDA).  We  discuss  each  of 
these  considerations  and  suggest  additional  steps  the  USDA  could 
take  to  address  them. 

PREPARING  FOR  THE  REVIEW 

Our  interviews  in  the  states  emphasized  that  preparing  for  the  review 
itself  often  generates  positive  spillovers.  Several  states  ran  seminars, 
attended  by  representatives  from  large  numbers  of  SFAs,  to  help 
them  prepare  for  their  SMI  reviews.  The  impending  SMI  reviews 
forced  SFA  staff  to  focus,  in  general,  on  nutritional  issues  and,  in 
particular,  on  the  DGA,  and  considerable  nutritional  education  of 
SFA  staff  appears  to  have  occurred. 

Furthermore,  preparing  for  the  reviews  itself  probably  improved 
procedures  and  consciousness  of  nutritional  issues.  In  some  SFAs, 
the  SMI  requirement  induced  a  movement  to  nutrient-based  menu 
planning  systems.  Even  in  the  majority  of  SFAs  that  have  not  made 
the  move  to  nutrient-based  menu  planning  systems,  preparing  for 
the  SMI  reviews  forced  changes  in  procedures.  Detailed  written 
menus  needed  to  be  developed.  Detailed  nutritional  information 
needed  to  be  obtained  from  suppliers. 

These  efforts  themselves  are  likely  to  result  in  improved  nutritional 
content  of  meals  served.  Proper  menus  and  nutritional  information 
for  ingredients  are  prerequisites  for  nutritional  improvement.  The 
increased  attention  to  the  nutritional  content  of  meals  is  itself  likely 
to  have  positive  effects  on  the  nutritional  content  of  meals  served  in 
the  future. 

RESULTS  OF  THE  REVIEW 

The  primary  mechanism  through  which  reviews  are  likely  to  affect 
future  meals  served  is  through  the  results  of  the  reviews.  First,  state 
officials  reported  that  both  they  and  SFA  officials  were  often  sur¬ 
prised  by  the  results  of  the  reviews.  Formal  nutrient  analysis  often 
yielded  results  very  different  than  those  expected  by  trained  nutri- 
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tionists  conducting  informal  reviews  of  menus  and  procedures.  State 
officials  reported  that  on  seeing  review  results,  SFA  officials  often  re¬ 
solved  to  improve  the  nutritional  content  of  their  meals. 

Furthermore,  the  reported  formal  “Correction  Plan”  is  specifically  in¬ 
tended  to  improve  the  nutritional  content  of  meals  served.  State  of¬ 
ficials  meet  with  SFA  officials  to  consider  the  results  of  the  SMI  re¬ 
view,  to  specify  changes  in  SFA  policy  (e.g.,  menus,  ingredients, 
suppliers,  cooking  procedures)  to  improve  the  nutritional  content  of 
me^s  served,  and,  ideally,  to  bring  them  into  compliance  with  DGA. 

The  interval  between  reviews  is  long  enough — about  five  years— that 
improvements  in  nutrition  are  not  immediately  necessary.  Failure  to 
change  procedures  as  specified  in  the  “Correction  Plan”  will  proba¬ 
bly  not  be  detected,  yet  willful  noncompliance  seems  unlikely. 
Almost  everyone  in  the  system  would  like  to  improve  the  nutritional 
content  of  meals.  However,  staff  changes,  changes  in  menus  and  the 
nutritional  content  of  ingredients,  and  different  priorities  (e.g.,  cost, 
staffing)  compete  for  the  attention  of  SFA  management.  Despite 
these  concerns,  it  seems  likely  that  the  results  of  the  reviews  and  the 
“Correction  Plan”  itself  will  improve  the  quality  of  future  meals 
served. 

QUALITY  OF  THE  REVIEW  INFORMATION 

Granted  these  positive  effects  of  preparing  for  the  SFA  reviews  and  of 
changed  procedures  in  response  to  the  results  of  these  reviews,  the 
primary  focus  of  this  report  has  been  on  procedures  for  collecting  the 
results  of  the  reviews  and  forwarding  them  to  USDA.  For  that  pur¬ 
pose,  SMI  reviews  are  only  as  good  as  the  quality  of  underlying  nutri¬ 
ent  analysis.  However,  staff  in  many  states  questioned  the  quality  of 
the  reviews.  In  particular,  they  stated  that  the  results  of  reviews 
would  vary  depending  on  who  did  the  review. 

Such  concerns  about  inter-rater  reliability  are  common  in  such  pro¬ 
grams  (e.g.,  SSA  disability  reviews;  see  Parsons,  1991,  and  the  litera¬ 
ture  it  spawned)  and  are  likely  to  be  particularly  salient  in  a  new  pro¬ 
gram  such  as  SMI.  It  is  usually  nearly  impossible  to  eliminate  such 
reliability  problems.  Nevertheless,  standard  steps  exist  to  minimize 
reliability  problems.  Such  efforts  usually  begin  with  the  promulga¬ 
tion  of  common  procedures  for  how  to  do  a  review  and  how  to  han- 
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die  standard  problems.  They  then  proceed  to  formal  initial  and  on¬ 
going  training  in  the  common  procedures.  Finally,  a  program  to  test 
the  quality  of  the  reviews  can  be  put  in  place  (i.e.,  re-reviewing  a 
sample  of  completed  reviews).  Depending  on  the  importance  and 
available  funding,  initiatives  to  address  concerns  about  inter-rater 
variation  may  be  worth  pursuing. 

REPRESENTATIVENESS  OF  REVIEW  AT  THE  SFA  LEVEL 

Even  if  it  were  assumed  that  the  reviews  are  perfect,  it  is  not  clear 
how  informative  the  results  of  a  review  (of  meals  served  one  given 
week  in  a  given  school)  are  about  all  the  meals  served  in  that  school 
(or  SFA).  For  example,  if  the  reviewed  meals  meet  the  guidelines, 
does  that  imply  that  all  meals  served  by  the  SFA  meet  the  guidelines? 
Conversely,  if  the  reviewed  meals  do  not  meet  the  guidelines,  does 
that  imply  that  no  meals  meet  the  guidelines?  The  distinction  is  im¬ 
portant.  Nutrition  is  a  result  of  all  meals  served,  not  the  meals  that 
were  reviewed.  Put  differently,  if  the  results  of  an  SFA’s  review  were 
to  become  publicly  available  at  an  SFA  level,  is  it  reasonable  for  par¬ 
ents  to  complain?  Or  rest  assured? 

The  minimum  required  SMI  effort  reviews  only  a  very  small  sample 
of  meals  served,  the  reviews  take  place  only  every  five  years,  and  only 
one  school  in  each  selected  SFA  is  reviewed  for  each  menu  plan  and 
age  group.  Within  that  school,  only  one  week’s  meals  are  reviewed. 
The  choice  of  schools  and  weeks  is  not  random  hut  rather  is  done  at 
the  convenience  of  the  reviewers.  There  are  serious  questions  about 
the  usefulness  of  this  information.  If  every  school  in  an  SFA  always 
served  the  same  meals  all  the  time,  in  exactly  the  same  way,  using 
exactly  the  same  ingredients  and  suppliers,  then  one  SMI  review 
would  be  enough.  That  level  of  uniformity  is,  of  course,  not  present. 
Instead,  it  is  quite  plausible  that  the  variation — across  years,  across 
schools,  across  weeks — is  so  large  that  the  results  of  one  review  pro¬ 
vide  little  information  about  the  nutritional  content  of  the  meals 
consumed  in  that  district.  If  so,  then  any  SFA-level  reports  to  USDA 
have  little  information  at  the  SFA-level  and  are  thus  not  useful  for 
parents. 

The  amount  of  variation — across  years,  across  schools,  across 
weeks — and  therefore  the  usefulness  of  SFA-level  data  is,  in  princi¬ 
ple,  an  empirical  question.  To  address  this  issue,  USDA  could  fund  a 
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small  pilot  study.  Such  a  pilot  study  would  randomly  select  a  small 
group  of  SFAs  (perhaps  a  few  dozen).  In  the  selected  SFAs,  the  pilot 
study  would  collect  SMI  type  information  across  schools  and 
through  time.  From  these  reviews,  it  would  be  possible  to  character¬ 
ize  the  ability  of  an  individual  SMI  review  to  provide  information 
about  the  nutritional  content  of  all  meals  provided  by  the  SFA. 

REPRESENTATIVENESS  OF  REVIEW  AT  THE  STATE  LEVEL 

The  USDA  Strategic  Plan  requires  information  about  all  meals  served. 
For  the  present  purposes,  USDA  is  only  seeking  information  on  the 
meals  that  are  reviewed.  Review  processes  can  only  collect  informa¬ 
tion  about  meals  reviewed.  Reviews  are  expensive;  reviewing  a  large 
number  of  meals  could  be  very  expensive.  However,  under  plausible 
assumptions  and  procedures,  reviews  of  a  small  number  of  meals  are 
informative  about  all  meals  served. 

If  each  review  is  properly  done,  then  standard  arguments  from  sta¬ 
tistical  sampling  imply  that  we  can  construct  a  statistically  accurate 
portrait  of  all  meals  served  by  a  group  of  SFAs  (e.g.,  a  state)  by 
averaging  results  across  multiple  reviews  and  across  individual 
weeks  from  individual  schools  from  multiple  districts.  This  would  be 
true  despite  the  fact  that  no  individual  review  provides  a  complete 
characterization  of  the  meals  provided  by  an  SFA. 

For  this  standard  statistical  sampling  argument  to  apply,  however,  it 
is  necessary  for  standard  random  sampling  assumptions  to  apply. 
For  this  situation  that  means: 

1.  Random  assignment  of  which  SFAs  are  to  be  reviewed  in  a  given 
year  (e.g.,  perhaps  random  assignment  to  each  of  the  five  years  of 
the  review  cycle). 

2.  Random  assignment  of  which  schools  within  the  SFA  are  to  be  re¬ 
viewed;  otherwise,  SFAs  might  pick  the  schools  with  the  '‘best” 
nutritional  content. 

3.  Random  assignment  of  which  week  during  the  school  year  reviews 
might  occur;  otherwise,  the  timing  of  the  review  might  be  corre¬ 
lated  with  particularly  good  or  bad  meals  (e.g.,  around  holidays  or 
the  beginning  or  ending  of  the  school  year). 
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Furthermore,  the  review  week  must  be  a  surprise.  Otherwise,  an  SFA 
might  substitute  a  “better"  (more  nutritious)  menu  for  the  review 
week. 

In  addition,  if  USDA's  goal  is  to  construct  a  statistically  precise  esti¬ 
mate  of  all  meals  served,  then  standard  sampling  arguments  suggest 
a  different  review  schedule.  Current  regulations  require  one  review 
per  SFA  in  each  five-year  cycle,  regardless  of  the  size  of  the  district. 
SFAs,  however,  vary  greatly  in  their  size.  At  the  extreme,  about  half  of 
the  meals  in  New  York  State  are  served  by  the  New  York  City  public 
school  system.  Statistical  considerations  suggest  allocating  more  re¬ 
sources  (i.e.,  more  reviews,  perhaps  more  than  every  five  years)  to 
such  large  SFAs,  and  fewer  resources  (i.e.,  fewer  reviews,  perhaps  less 
frequently  than  every  five  years)  to  smaller  SFAs.  Estimating  how 
much  more  would  require  the  results  of  a  pilot  study  like  the  one  dis¬ 
cussed  above. 

Such  an  approach  has  a  precedent:  Current  CRE  regulations  already 
promote  more  intensive  efforts  in  larger  districts.  Under  the 
Expanded  Review  Cycle,  “State  Agencies  are  encouraged  to  conduct 
administrative  reviews  of  large  school  food  authorities  and  of  any 
school  food  authorities  which  may  benefit  from  a  more  frequent  in¬ 
terval  than  the  5 -year  cycle  required  by  paragraph  (c)  (1).”^ 

Adjusting  the  intensity  of  the  review  schedule  with  the  size  of  the  SFA 
is  also  attractive  from  a  burden  perspective.  Many  state  officials  ex¬ 
pressed  concerns  about  the  burden  of  SMI  reviews  for  small  SFAs. 
For  small  SFAs,  food  service  is  a  small-scale  operation.  Nebraska 
suggested  that  FNS  should  accept  data  aggregated  by  school  size, 
since  they  have  so  many  very  small  SFAs.  In  Texas,  50  percent  of  the 
students  are  in  46  SFAs  and  the  other  50  percent  are  spread  out  over 
more  than  1,000  SFAs.  According  to  the  state  staff,  many  of  these 
small  SFAs  feel  that  SMI  is  beyond  them  because  they  do  not  have 
the  necessary  education  or  expertise.  In  most  small  SFAs,  staff  have 
little  nutritional  training,  and  the  cook  is  often  the  only  food  service 
staff  person.  In  such  situations,  SMI  reviews  are  a  great  burden.^ 


^Paragraph  (c)(1)  states  that  “at  a  minimum,  State  Agencies  shall  conduct  administra¬ 
tive  reviews  of  all  school  food  authorities  at  least  once  during  each  5-year  review  cy¬ 
cle,”  210  use  18. 

^These  might  also  be  the  SFAs  whose  meals  have  particularly  poor  nutritional  content. 
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State  officials  expressed  concern  that  SFAs  would  pull  out  of  the 
program  once  they  knew  FNS  would  require  the  collection  of  nutri¬ 
tional  content. 

USDA  has  already  partially  moved  toward  lessening  this  burden  by 
allowing  less-intensive  review  of  RCCIs.^  Careful  consideration  of 
less-intensive  review  for  small  SFAs  might  be  worthwhile.  Exempting 
small  SFAs — under  1,000  students — would  exclude  43  percent  of  the 
SFAs,  but  only  5  percent  of  the  students  from  SMI.^  Eliminating  (or 
at  least  scaling  back)  review  of  these  small  SFAs  might  lower  resis¬ 
tance  to  the  program  and  the  burden  of  the  unfunded  mandate  on 
the  state  authorities.  The  cost  in  the  completeness  of  the  data  would 
be  small. 

CONCLUSION 

This  final  chapter  has  reviewed  broader  issues  relating  to  the  SMI 
program.  We  discussed  the  positive  effects  of  SMI  reviews  arising 
from  efforts  to  prepare  for  the  review  and  from  changes  in  meals  due 
to  the  results  of  the  review. 

This  report  focused  on  how  states  should  report  the  results  of  SMI 
reviews  to  USDA.  For  that  purpose,  the  quality  and  representative¬ 
ness  of  the  underlying  review  results  are  crucial.  We  presented  three 
concerns  about  current  procedures  and  suggested  some  actions 
USDA  could  undertake  to  address  these  concerns. 

USDA  is  aware  of  these  concerns,  and  similar  issues  arise  in  the  CRE 
process.  The  SMI  review  process  is  still  in  its  first  cycle.  Now,  as  the 
first  cycle  winds  down  and  as  USDA  considers  how  states  should  re¬ 
port  SMI  results,  is  an  appropriate  time  to  consider  the  research  ac¬ 
tivities  and  operational  changes  discussed  in  this  chapter. 

However,  it  is  important  to  note  the  concerns  expressed  by  the  states 
about  the  effort  required  to  perform  the  SMI  reviews.  How  to  pro¬ 
ceed  with  the  review  program  requires  balancing  the  direct  effects  on 
the  nutritional  content  of  meals,  the  utility  of  the  aggregated  data 


^FNS  Memorandum  #99-5;  1999. 

"^The  Gallup  Organization  and  PROMAR  International,  1999. 


52  State  Monitoring  of  NSLP  Nutritional  Content 


(given  the  quality  of  the  underlying  reviews),  and  the  statutory  re¬ 
quirement  against  the  required  effort  by  state  and  SFA  staff.  Staff  in 
several  states  argued  strongly  that  they  did  not  believe  the  effort  was 
worthwhile  given  the  perceived  benefit. 


Appendix  A 

PROTOCOL  FOR  DATA  COLLECTION 


Date:  _ 

Name:  _ 

Agency:  _ 

Title:  _ 

Phone  Number: 

Name:  _ 

Agency:  _ 

Title:  _ 

Phone  Number: 

Name:  _ 

Agency:  _ 

Title:  _ 

Phone  Number: 

Name:  _ 

Agency:  _ 

Title:  _ 

Phone  Number: 
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FNS  hopes  to  ultimately  be  able  to  aggregate  nutrient  analysis 
information  to  the  state  and  national  level.  To  do  this,  it  needs 
certain  data  elements,  and  these  data  elements  have  to  be  in 
electronic  format  for  each  School  Food  Authority  (SFA)  in  the  state. 
This  project  will  look  at  what  seven  states  are  currently  doing  and 
what  data  elements  are  being  collected.  FNS  does  not  want  us  to 
recommend  major  changes  to  the  way  that  your  state  is  currently 
collecting  the  SMI  information.  Instead,  we  will  be  looking  at  what 
you  are  doing  and  what  minor  changes  you  could  make  in  order  to 
get  FNS  what  it  needs. 

RAND  plans  to  adhere  to  the  following  timeline  for  this  project.  As 
you  can  see,  there  is  quite  a  bit  of  time  for  comments  from  FNS.  FNS 
plans  to  share  the  preliminary  results  with  the  states  and  give  the 
states  an  opportunity  to  comment  on  the  findings  before  RAND 
completes  any  of  the  memos  or  reports. 


TIMELINE  FOR  USDA  CNAP  TASK  #4 


School  Meal  Program  Government  Performance  and 


Results  Act  of  1993  Standards 


Item /Deliverable 

Tentative  Date 

Data  collection  protocols  for  state  visits  finalized 

5/21/99 

Fieldwork  begins 

6/1/99 

Fieldwork  ends 

9/1/99 

First  memo  draft  delivered  to  FNS 

10/1/99 

FNS  provides  comments  to  RAND 

10/22/99 

First  memo  revised  delivered  to  FNS 

11/5/99 

FNS  provides  comments  to  RAND 

11/26/99 

Second  memo  draft  delivered  to  FNS 

12/24/99 

FNS  provides  comments  to  RAND 

1/21/00 

Second  memo  revised  delivered  to  FNS 

2/4/00 

FNS  provides  comments  to  RAND 

2/25/00 

Report  draft  delivered  to  FNS 

3/24/00 

FNS  provides  comments  to  RAND 

5/19/00 

Final  report  text  delivered  to  FNS 

6/9/00 

Final  report  documents  delivered  to  FNS 

8/11/00 

Protocol  for  Data  Collection  55 


BACKGROUND  QUESTIONS 

1 .  How  many  SFAs  are  there  in  your  state? 

_  SFAs  DK 

2,  How  do  you  define  an  SFA?  Is  it  a  school  district  or  something 
else? 


3.  When  did  your  state  begin  SMI  reviews? 

1996/1997  1997/1998  1998/1999  Haven’t  started  DK 

4.  Why  did  you  start  then  as  opposed  to  earlier  or  later? 


5.  How  many  SMI  reviews  were  completed  in 


1996/1997?  _ 

Reviews 

DK 

1997/1998?  _ 

Reviews 

DK 

1998/1999? 

Reviews 

DK 

6.  How  do  you  define  a  completed  review? 


7.  When  do  you  expect  to  complete  the  first  round  of  SMI  reviews? 

1998/1999  1999/2000  2000/2001 

2001/2002  2002/2003  DK 

8.  Do  you  think  your  state  will  need  to  make  any  changes  in  the 
future  to  process  or  staffing  in  order  to  complete  the  SMI 
reviews  in  5  years? 
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9.  Are  SMI  reviews  done  in  conjunction  with  CRE  reviews? 

Yes  No  DK 

10.  If  you  do  SMI  reviews  in  conjunction  with  CRE,  did  you  have  to 
add  staff  to  do  this?  What  kind  of  training  was  involved? 

11.  Do  your  reviewers  have  access  to  e-mail? 

Yes  No  DK 

12.  Do  your  reviewers  have  access  to  the  Internet? 

Yes  No  DK 

13.  Is  there  any  other  software  (i.e.,  MS  Excel)  that  reviewers  use  for 
completing  reviews?  If  so,  what?  Is  the  same  software  used  at 
the  state  level  and  at  the  SEA  level? 

Yes  No  DK 

14.  How  many  people  are  involved  in  doing  SMI  reviews  and 
analysis? 

_ People 

15.  Where  are  they  located?  For  whom  do  they  work? 


16.  What  are  each  of  their  roles  in  the  SMI  reviews? 
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17.  Does  the  state  agency  have  access  to  the  nutrient  analysis 
information? 

Yes  No  DK 

18.  Do  you  feel  that  the  SMI  reviews  are  necessary  to  bring  school 
meals  into  compliance  with  the  Dietary  Guidelines  for 
Americans  and  the  Recommended  Daily  Allowances? 

Yes  No  DK 

Comments: 


19.  Do  you  think  it  would  be  difficult  for  the  reviewers  in  this  state 
to  provide  information  to  FNS  directly? 


Yes 

Comments: 


No 


DK 
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PROCESS  IN  THE  STATE 

We  are  trying  to  create  a  flowchart  that  shows  how  and  where  the 
information  is  collected,  whom  the  information  is  passed  on  to, 
when  and  how  the  information  is  revised,  and  what  format  the 
information  is  in  at  every  stage.  I  would  like  to  go  through  the 
process  separately  for  an  SFA  using  a  food-based  system  and  an  SFA 
using  a  nutrient-based  system,  since  these  processes  are  likely  to  be 
different.  Bear  in  mind  that  FNS  will  require  you  to  provide  the  data 
it  requests  for  both  food-based  and  nutrient-based  systems. 

1.  Who  collects  the  raw  information  for  nutrient  analyses  on  food 
offered  in  any  given  SFA? 


2.  With  what  organization  is  this  person  employed? 


3.  In  what  format  is  the  information  collected?  Does  this  format 
change  over  the  course  of  the  review?  For  example,  if  the  initial 
information  is  collected  on  hard  copy,  is  it  ever  converted  to  an 
electronic  version? 


4.  Who  performs  the  nutrient  analysis  of  this  information? 
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5.  Are  there  any  steps  between  the  initial  collection  of  information 
and  the  nutrient  analysis?  If  so,  what  are  they  and  who  performs 
them? 


6.  Is  the  nutrient  analysis  ever  revised  after  it  is  initially  performed? 
If  so,  when  and  by  whom?  Where  is  the  revision  information 
recorded? 


7.  How  often  are  nutrient  analyses  usually  revised? 


8.  Are  data  elements  ever  added  or  deleted  from  the  information 
during  this  process? 


9.  Is  the  information  aggregated  in  some  way  other  than  at  the 
state  level?  For  example,  at  a  district  or  regional  level?  If  so,  at 
what  level? 
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10.  Where  are  SFA-level  records  kept  and  in  what  format? 


1 1 .  When  is  the  information  sent  to  the  state  and  by  whom? 


12.  How  is  the  information  sent  to  the  state?  Electronically?  Hard 
copy?  If  electronically,  please  describe  the  protocols  used.  For 
example,  is  the  information  recorded  on  a  diskette?  Sent  by  e- 
mail?  Other? 


13.  Where  is  the  information  kept  and  in  what  format?  If  applicable, 
please  give  a  name  and  telephone  number  for  the  person  who 
would  have  this  information  at  the  state  level. 
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14.  Does  the  state  do  an  independent  nutrient  analysis  for  SMI 
reviews,  or  does  the  state  review  existing  analyses,  or  both?  In 
which  cases  does  the  state  do  independent  reviews?  In  which 
cases  does  the  state  review  nutrient  analyses  performed 
elsewhere?  (Keep  in  mind  that  it  is  possible  for  the  state  to  do 
both  if  it  reviews  nutrient  analyses  done  by  an  SFA  using  a 
NuMenus  or  Assisted  NuMenus  system  but  does  the  actual 
nutrient  analyses  for  SFAs  using  food-based  systems.) 


15.  At  what  point  would  it  be  best  to  have  the  nutrient  analysis 
information  sent  to  FNS?  From  the  state?  From  the  reviewers 
themselves?  Why? 


16.  Which  of  these  steps,  if  any,  would  need  to  change  in  order  to 
meet  FNS’s  goals  for  the  selected  data  elements  being  sent  in 
electronic  format? 


17.  It  is  possible  that,  in  the  future,  FNS  may  be  able  to  negotiate 
with  the  companies  that  have  created  the  nutrient  analysis 
software  to  add  a  function  where  you  would  be  able  to  create  the 
report  for  FNS  right  from  the  software.  If  that  were  to  happen, 
what  changes  would  you  need  to  make  to  your  current 
procedure? 
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18.  Which  of  the  required  data  elements  are  currently  missing  from 
the  software  package  you  are  using  for  your  nutrient  analyses? 


19.  If  the  software  companies  do  not  agree  to  add  this  function,  how 
will  you  incorporate  the  additional  elements  into  an  electronic 
report  for  FNS? 


20.  Do  you  think  that  there  are  any  data  elements  that  should  be 
added  to  or  deleted  from  the  list  to  send  to  FNS?  If  so,  which 
items  and  why? 


21.  Do  you  have  any  opinion  on  sending  the  information  to  FNS 
electronically?  Will  this  create  any  problems  for  your  state? 


22.  How  often  do  you  think  the  states  should  have  to  report  this 
information  to  FNS?  FNS  is  required  to  prepare  an  annual 
strategic  plan.  Therefore,  it  is  leaning  toward  annual  collection 
of  this  information.  Would  this  cause  problems  for  your  state? 
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23.  Can  you  think  of  any  alternatives  for  any  of  the  processes  we 
have  discussed  so  far? 


24,  FNS  would  like  us  to  solicit  comments  from  the  state  about  this 
process.  Do  you  have  anything  that  you  would  like  us  to  pass  on 
to  them? 
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DATA  ELEMENTS 

Based  on  conversations  with  FNS,  the  following  data  elements  will  be 
necessary  in  order  to  aggregate  the  information  at  the  state  or 
national  level  in  a  way  that  will  be  useful.  The  data  elements  will 
include  both  the  measures  of  the  eight  quantitative  and  three 
qualitative  standards  for  each  SFA,  as  well  as  the  standards 
themselves.  These  standards  are  necessary  because  FNS  may  not 
have  access  to  the  standards  an  SFA  is  using,  especially  if  it  is  using  a 
nontraditional  USDA- approved  system. 

In  the  "Type"  column,  C  indicates  that  a  character  response  is 
required;  N  indicates  that  a  numeric  response  is  required. 


Element _ 

Unique  reference  code 
for  analysis 


Name  of  SFA 

Contact  Person  for 
Analysis 

Contact  Person  Phone  # 

Contact  Person  E-mail 

Contact  Person  Address, 
line  1 

Contact  Person  Address, 
line  2 

Contact  Person  Address, 
lines 

Contact  Person  City 

Contact  Person  State 

Contact  Person  Zip  Code 

Name  of  Person  Doing 
Review 

Name  of  Person  Doing 
Analysis 

Start  Date  of  Review 
Week 

End  Date  of  Review 
Week 


Reporting  Format 


J  (First) 
.1  (Last) 


UU!/UI_|/|_U-I-! 

(MM  /DD/YYYY) 

UU|/|_U/U_U-I 

(MM /DD/YYYY) 


Type _ Notes _ 

C  Unique  code  for 
FNS:  State  will 
not  repeat 
reference  code 
for  any  future  or 
past  analysis 
C 
C 

C 

N 

C 

C 

c 

c 


c 

N 

c 

c 

N 

N 
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Date  Analysis  Closed 

1 _ 1 _ 1  (Month) 

|_|_UI_1  (Year) 

N 

N 

State  in  Which  Review 

UUI 

C 

Was  Done 

Type  of  School  Food 
Authority 

1-1 

N 

l=Public  School 
District 

2=Private  School  or 
District 
3=Residential 

Child  Care 
Institution 
4=Other 

Number  of  Schools  in 

1  1  i  1  1 

N 

SFA 

Menu  Planning  System 
Used 

l-l 

C 

N  =  NuMenus 

A  =  Assisted 
NuMenus 

T  =  Traditional 
Food-Based 

E  =  Enhanced 
Food-Based 

0  =  Other  USDA- 
Approved 

System 

Clarification  of  Other 

USD  A- Approved 

System 

(free  form) 

C 

Define  Other 

USDA- 
Approved 
System(s)  Used 

Was  Analysis  for  Lunch 
Only  or  Lunch  and 
Breakfast? 

l-l 

c 

L  =  Lunch  Only 

B  =  Both  Lunch  & 
Breakfast 

Analysis  Weighted? 

!-l 

c 

Y==Yes,N  =  No 

Was  a  Plan  of  Correction 
Needed? 

l-l 

N 

0=No 

l=Yes 

Lowest  Grade /Age  Range 
Used  for  Analysis 

l-l 

c 

P  =  Pre- 

Kindergarten/ 

Invalid 

Response 

K  =  Kindergarten/ 
Invalid 

Response 

0  =  Invalid 

Response /Infant 

1  =  First  Grade/ 1st 
Year 

2  =  Second 

Grade/2ndYear 
(and  so  on) 
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Highest  Grade/Age  |_|  C 

Range  Used  for 
Analysis 


Antdysis  Done  by  Grade  | _ |  C 

Range  or  Age  Range 

**Average  Daily  Number  !_1__L|_|  N 

of  Lunches  Served 
During  Week  for 
Which  Analysis  Done 
Actutil  Content  from 
Analysis  Software 
Output: 


Caloric  content  (kcal)  N 

Fat  content  (g)  |_|_j .  |_|_|  N 

Saturated  Fat  content  (g)  j _ j _ j .  j _ | _ j  N 

Protein  content  (g)  l—l— I  •  i_i_|  N 

Calcium  content  (mg)  Li_i_| .  i_i_|  N 

Iron  content  (mg)  l—l—l  •  l—l—j  N 

Vitamin  A  content  (RE)  N 

Vitamin  C  content  (mg)  _ j .  |_| _ |  N 

Cholesterol  content  (mg)  j _ j _ j _ |  N 

Sodium  content  (mg)  N 

Fiber  content  (g)  j _ j _ j .  | _ | _ |  N 

Standards  for  Menu  Plan 
Used: 

Caloric  content  (kcal)  | _ | _ | _ | _ |  N 

Fat  content  (g)  |_|_| .  j_|_|  N 

Saturated  fat  content  (g)  i_i_j .  i_i_i  N 

Protein  content  (g)  j _ j _ j .  j _ | _ j  N 

Calcium  content  (mg)  j _ j _ N 

Iron  content  (mg)  l—l—i  •  I— 1_|  N 

Vitamin  A  content  (RE)  N 

Vitamin  C  content  (mg)  j _ j _ j .  | _ | _ |  N 

Comments  (free  form)  C 


P  =  Pre- 
Kindergarten/ 
Invalid 
Response 
K  =  Kindergarten/ 
Invalid 
Response 
0  =  Invalid 
Response/Infant 

1  =  First  Grade /1st 
Year 

2  =  Second 
Grade/2nd  Year 

(and  so  on) 

G  =  Grade  Range 
A  =  Age  Range 


Space  for  any 
clarification 
comments  from 
the  SFA 
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INDIVIDUALS  INTERVIEWED  IN  THE  SEVEN  STATES 


Table  B.l 

Individuals  Interviewed  in  California 

Individual 

Interviewed 

Agency 

Title 

Jean  Naylor 

State  Department  of  Education,  Field 

Child  Nutrition 

Services  Unit 

Supervisor  II 

Louise  Casias 

State  Department  of  Education,  Field 

Child  Nutrition 

Services  Unit 

Supervisor  I 

Violet  Henry 

State  Department  of  Education,  Field 
Services  Unit 

Child  Supervisor  I 

Valerie  Fong 

State  Department  of  Education,  School 

Staff  Services 

Nutrition  Programs  Unit 

Manager  II 

Jan  Barnhouse 

State  Department  of  Education,  School 

Child  Nutrition 

Nutrition  Standards  Unit 

Supervisor  I 

Andrew  Laufer 

State  Department  of  Education,  Field 

Child  Nutrition 

Services  Unit 

Supervisor  I 

Cindy  Schneider 

State  Department  of  Education,  School 

Child  Nutrition 

Nutrition  Standards  Unit 

Consultant 

Marily  Briggs 

State  Department  of  Education,  School 
Nutrition  Services  Division 

Director 
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Table  B.2 

Individuals  Interviewed  in  Georgia 


Individual 

Interviewed 

Agency 

Title 

Annette  Bomar 
Hopgood 

Judieth  Hunt 

Eugenia  Seay 

Georgia  Department  of  Education 

Georgia  Department  of  Education 

Georgia  Department  of  Education 

Director,  School  of 

Community 

Nutrition 

Education  Grant 
Program 

Consultant 

School  Nutrition 
Program  Manager 

Table  B.3 

Individuals  Interviewed  in  Nebraska 

Individual 

Interviewed 

Agency 

Title 

Connie 

Stefkovich 

Shawn  Voudracek 

Alisanne  Ells 

Mary  Ann 

Brennan 

Nebraska  Department  of  Education 

Nebraska  Department  of  Education 

Nebraska  Department  of  Education 

Nebraska  Department  of  Education 

Administrator- 
Nutrition  Services 
Consultant- 
Nutrition  Services 
Consultant- 
Nutrition  Services 
Consultant- 
Nutrition  Services 
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Table  B.4 

Individuals  Interviewed  in  New  Jersey 


Individual 

Interviewed 

Agency 

Title 

Kathy  Kuser 

Barbara  Guarnieri 

Joanne  Lontz 

Jill  Niglio 

Bureau  of  Child  Nutrition  Programs,  New 
Jersey  Department  of  Agriculture 

Bureau  of  Child  Nutrition  Programs,  New 
Jersey  Department  of  Agriculture 

Bureau  of  Child  Nutrition  Programs,  New 
Jersey  Department  of  Agriculture 

Bureau  of  Child  Nutrition  Programs,  New 
Jersey  Department  of  Agriculture 

State  Director 

Program 

Development 

Specialist  I-Public 

Program 

Development 

Specialist  I- 

Nonpublic 

Regional 

Coordinator- 

Central 

Table  B.5 

Individuals  Interviewed  in  New  York 

Individual 

Interviewed 

Agency 

Tide 

Frances 

O'Donnell 

Linval  Foster 

Debbie  Favro 

Sandy  Sheedy 

Child  Nutrition  Program  Administration, 
New  York  State  Education  Department 
Child  Nutrition  Program  Administration, 
New  York  State  Education  Department 
Child  Nutrition  Program  Administration, 
New  York  State  Education  Department 
Child  Nutrition  Program  Administration, 
New  York  State  Education  Department 

Coordinator 

Associate 

Assistant 

Nutrition  Program 
Representative 

Table  B.6 

Individuals  Interviewed  in  Texas 

Individual 

Interviewed 

Agency 

Title 

John  Perkins 

Debbie  Owens 

Texas  Education  Agency,  Child  Nutrition 
Programs 

Texas  Education  Agency,  Child  Nutrition 
Programs 

Director 

Program  Director 
for  Compliance 
and  Monitoring 

70  State  Monitoring  of  NSLP  Nutritional  Content 


Table  B.7 

Individuals  Interviewed  in  Wisconsin 


Individual 

Interviewed 

Agency 

Title 

Julie  Cox 

Department  of  Public  Instruction,  Food 

Child  Nutrition 

and  Nutrition  Services 

Program 

Consultant 

Carol  Philipps 

Department  of  Public  Instruction,  Food 

Program 

and  Nutrition  Services 

Coordinator 

BIBLIOGRAPHY 


Abt  Associates.  “Evaluation  of  the  Nutrient  Standard  Menu  Planning 
Demonstration:  Findings  from  the  Formative  Evaluation.” 
February  1997. 

Burghardt,  J.,  A.  Gordon,  N.  Chapman,  P.  Gleason,  and  T.  Fraker.  The 
School  Nutrition  Dietary  Assessment  Study:  School  Food  Service, 
Meals  Offered,  and  Dietary  Intakes,  FNS/USDA,  October  1993. 

Burghardt,  John,  Anne  R.  Gordon,  and  Barbara  Devaney.  “Dietary 
Effects  of  the  National  School  Lunch  Program  and  the  School 
Breakfast  Program.”  American  Journal  of  Clinical  Nutrition, 
61:1(S),  January  1995,  pp.  221S~231S. 

Cleveland,  L.,  et  al.  “Pyramid  Servings  Data:  Results  from  USDA’s 
1994  Continuing  Survey  of  Food  Intakes  by  Individuals.”  Food 
Surveys  Research  Group,  March  1997.  Available  on-line  at 
http://www.barc.usda.gov/bhnrc/foodsurvey/home.htm. 

Coordinated  Review  Effort  Data  Report,  School  Year  1994-1995: 
National  Summary,  Food  and  Consumer  Service,  USDA. 

Dietary  Guidelines  Advisory  Committee.  Report  of  the  Dietary 
Guidelines  Advisory  Committee  on  the  Dietary  Guidelines  for 
Americans,  1995.  Available  on-line  at  http://www.nal.usda.gov/ 
fnic  /  Dietary  /  dietg95  .htm. 

Federal  Register,  60:113,  Tuesday,  June  13, 1995. 

Fisher,  J.,  and  L.  Birch.  “Restricting  Access  to  Palatable  Foods  Affects 
Children’s  Behavioral  Response,  Food  Selection  and  Intake,” 


71 


72  State  Monitoring  of  NSLP  Nutritional  Content 


American  Journal  of  Clinical  Nutrition,  69:6,  June  1999,  pp.  1264- 
1272. 

FNS  Memorandum  #99-5:  Nutrition  Reviews  of  Residential  Child 
Care  Institutions  (sent  to  Regional  Directors  on  April  28, 1999  from 
Stanley  C.  Garnett,  Director,  Child  Nutrition  Division). 

The  Gallup  Organization  and  PROMAR  International.  School  Meals 
Initiative  Implementation  Study:  First  Year  Report  (Draft  Report); 
June  1, 1999. 

General  Accounting  Office.  “School  Lunch  Program:  Cafeteria 
Managers’  Views  on  Food  Wasted  by  Students,”  Letter  Report,  July 
18,  1996,  GAO/RCED-96-191. 

National  School  Lunch  Act,  42  USC  1751. 

Osborne,  David,  and  Ted  Gaebler.  Reinventing  Government:  How 
the  Entrepreneurial  Spirit  Is  Transforming  the  Public  Sector. 
Reading,  Mass.:  Addison  Wesley,  1993,  p.  146. 

Parsons,  Donald.  “Self-Screening  in  Targeted  Public  Transfer 
Programs.”  Journal  of  Political  Economy,  99, 1991,  pp.  859-876. 

Subar,  A.,  S.  Krebs-Smith,  A.  Cook,  and  L.  Kahle.  “Dietary  Sources  of 
Nutrients  among  U.S.  Children,  1989-1991.”  Pediatrics,  102:4, 
October  1998. 

USDA  Center  for  Nutrition  Policy  and  Promotion.  The  Healthy 
Eating  Index,  October  1995. 


